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This  dissertation  addresses  the  issue  of  applying  game  theoretic  controllers  to  the 
multiobjective  linear  regulator  problems.  The  classical  regulator  problem  in  optimal  control 
theory  is  based  on  the  assumption  that  feedback  controllers  optimize  a linear  system  with 
a single  integral  performance  index.  For  the  general  multiple  input/multiple  output  linear 
system,  there  may  exist  more  than  one  performance  index.  Differential  game  theory,  which 
frequently  deals  with  multiple  performance  indices,  is  found  to  be  directly  applicable  for  the 
multiobjective  control  problems.  This  observation  motivates  this  research  to  design  game 
theoretic  controllers  for  the  multiobjective  linear  regulator  problems. 

There  are  two  ways  to  design  game  theoretic  controllers  in  the  multiobjective  linear 
regulator  problem.  For  a given  set  of  weighting  matrices  associated  with  the  integral 
performance  indices,  the  feedback  gain  matrix  can  be  obtained  by  solving  coupled  algebraic 
Riccati  equations.  Most  up-to-date  coupled  algebraic  Riccati  equation  solvers  are  based  on 
an  iterative  Newton  method.  It  is  well  known  that  convergence  for  such  an  iterative  scheme 
strongly  depends  on  the  initial  guesses.  In  the  first  part  of  this  research,  a robust  integrative 
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scheme  based  on  homotopy  theory  is  developed.  Current  homotopy  schemes  can  generate 
solutions  for  algebraic  Riccati-type  equations;  however,  the  solutions  are  not  guaranteed  to 
be  positive  definite.  For  a standard  algebraic  Riccati  equation,  the  proposed  homotopy 
scheme  generates  the  positive  definite  solution,  which  is  the  only  solution  of  interest  in 
control  problems.  To  make  the  homotopy  scheme  more  efficient,  the  symmetric  property 
of  the  solution  of  an  algebraic  Riccati-type  equation  is  used  in  conjunction  with  the 
Kronecker  sum. 

In  the  second  part  of  this  research,  a new  optimal  pole  placement  scheme  based  on  the 
differential  game  theory  is  developed  for  a linear  time  invariant  system  with  prescribed 
closed-loop  pole  locations.  The  pole  placement  scheme  is  capable  of  shifting  a single  pole 
or  a pair  of  poles  to  the  desired  location(s),  while  providing  “additional”  design 
specifications  for  the  linear  system.  Multiple  pole  shifting  tasks  can  be  implemented  by 
utilizing  the  shifting  algorithm  for  a single  pole  or  a pair  of  poles  recursively.  It  is 
demonstrated  that  the  proposed  scheme  expands  the  design  domain  from  classical  optimal 
control  scenario  and  allows  additional  design  criteria  to  be  imposed  on  the  closed-loop 
system. 
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CHAPTER  1 
INTRODUCTION 

1.1  Motivation 

The  classical  regulator  problem  in  optimal  control  theory  is  based  on  the  assumption 
that  state  feedback  controllers  optimize  the  performance  of  a linear  system  based  on  a single 
integral  quadratic  performance  index.  However,  in  many  real  dynamical  problems,  it  may 
not  be  sufficient  to  describe  the  multiple  objectives  of  a multiple  input/multiple  output  linear 
system  with  a single  integral  quadratic  performance  index  [1,2].  For  example,  consider  the 
communication  satellite  shown  in  Fig.  1.1.  On  one  hand,  the  “communication  payload”  is 
maneuvered  to  point  to  earth  for  the  purpose  of  data  transmission.  On  the  other  hand,  the 
“energy  payload”  must  be  oriented  such  that  the  on-board  solar  panel  can  absorb  the 
maximum  amount  of  solar  energy  [1],  These  are  conflicting  requirements  which  must  be 
addressed  by  the  on-board  controllers.  Another  example  of  a multiple  objective  dynamical 
system  is  the  United  States  Microgravity  Payload  (USMP),  which  uses  the  unique  attributes 
of  the  space  shuttle  cargo  bay  to  conduct  large,  sophisticated  microgravity  experiments.  In 
the  USMP  program,  the  space  shuttle  is  maneuvered  to  accomplish  its  mission  in  the  space. 
At  the  same  time,  the  on-board  microgravity  payload  must  be  isolated  from  the  space 
shuttle’s  motion  in  order  to  perform  dedicated  scientific  experiments  such  as  crystal  growth. 

In  general,  the  strategic  interactions  among  all  controllers  for  such  multiobjective 
multiple  input/multiple  output  systems  cannot  be  properly  described  by  the  classical  optimal 
control  theory  which  only  deals  with  a single  optimization  objective.  For  most  dynamical 
systems,  further  analysis  on  the  multiobjective  linear  and/or  nonlinear  system  should  be 
further  developed.  A differential  game  theory,  which  is  the  study  of  interacting  strategies 
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for  multiple  intelligent  players,  is  found  to  be  directly  applicable  for  these  multiobjective 
control  problems. 


It  has  been  decades  since  the  initial  model  of  the  differential  game  in  a linear  system 
was  proposed  by  Isaacs  [3].  Starr  and  Ho  [4]  proposed  a nonzero-sum  noncooperative 
differential  game  with  a given  set  of  multiple  integral  performance  indices  for  the  general 
n-player  game.  The  weighting  matrices  in  those  integral  performance  indices  are  chosen  in 
advance.  In  the  finite  time  case,  the  time  varying  solutions  to  an  n-player  nonzero-sum 
noncooperative  differential  game  can  be  obtained  by  solving  n-coupled  differential  Riccati 
equations.  In  the  infinite  time  horizon  case,  the  time  invariant  solution  is  obtained  by  solving 
n-coupled  algebraic  Riccati  equations. 

However,  due  to  its  mathematical  complexity,  the  majority  of  current  research  has  been 
restricted  to  the  solution  of  a 2-player,  nonzero-sum,  noncooperative,  infinite  time  horizon 
differential  game.  Most  solution  schemes  currently  in  use  are  based  on  Newton’s  iterative 
method  [1,5,6].  Another  scheme,  based  on  a conjugate  gradient  technique  was  formulated 


Solar  Energy 


Figure  1.1  Communication  Satellite 
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by  Innocenti  and  Schmidt  [7],  However,  the  convergence  to  the  desired  solutions  of  all 
iterative  schemes  strongly  depends  on  the  choice  of  the  initial  guesses,  even  for  the  2-player 
differential  game. 

Besides  the  numerical  difficulty  of  solving  the  general  n-player  nonzero-sum, 
noncooperative  differential  game,  the  most  ambiguous  issue  to  a control  engineer  is  the 
selection  of  weighting  matrices  in  those  performance  indices.  Different  combinations  of 
weighting  matrices  will  lead  to  entirely  different  closed-loop  pole  locations;  thus  completely 
different  performance  for  the  closed-loop  system  responses.  To  the  best  of  the  author’s 
knowledge,  the  issue  of  weighing  matrices  selection  for  game  theoretic  state  feedback 
controllers  with  prescribed  closed-loop  poles  has  never  been  addressed  in  the  literature. 

1.2  Research  Scope 

In  this  research,  two  methodologies  for  designing  linear  time  invarient,  game  theoretic, 
state  feedback  controllers  in  the  regulator  problem  are  discussed.  First,  for  a given  set  of 
weighting  matrices  for  integral  quadratic  performance  indices,  the  state  feedback  gain 
matrix  can  be  obtained  by  solving  coupled  algebraic  Riccati  equations.  Most  up-to-date 
coupled  algebraic  Riccati  equations  solvers  are  based  on  Newton’s  iterative  method.  It  is 
well  known  that  convergence  for  such  iterative  schemes  strongly  depend  on  the  initial 
guesses.  A bad  choice  of  an  initial  guess  may  result  in  divergence  for  the  iterative  scheme. 
To  overcome  such  numerical  difficulties,  an  integrative  scheme  based  on  homotopy  theory 
is  proposed  in  the  first  part  of  this  research. 

A homotopy  scheme  is  a path  following  algorithm,  which  “bends”  a known  solution 
into  the  desired  solution  [8],  The  homotopy  path  is  described  by  the  homotopy  differential 
equation.  For  an  algebraic  Riccati-type  equation,  the  associated  homotopy  differential 
equation  is  in  the  form  of  a Lyapunov-type  equation.  The  proposed  homotopy  scheme  has 
two  features: 
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( 1 ) .  In  comparison  with  generic  homotopy  codes,  the  developed  homotopy  algorithm 

is  more  efficient  in  solving  coupled  algebraic  Riccati  equations.  This  improvement 
in  efficiency  resulted  from  exploiting  the  symmetric  property  of  coupled  algebraic 
Riccati  equations  and  the  use  of  the  Kronecker  sum. 

(2) .  For  standard  algebraic  Riccati  equations,  the  current  homotopy  schemes  [9,10] 

may  not  result  in  positive  definite  solutions.  A homotopy  scheme  is  developed 
which  guarantees  that  the  solution  is  positive  definite;  this  is  the  solution  of 
interest  in  the  linear  quadratic  regulator  problems. 

An  inverse  methodology  for  designing  a linear  time  invariant  state  feedback  controller 
is  defined  as  follow:  “given  a set  of  closed-loop  pole  locations  and  input  weighting  matrices 
in  the  integral  quadratic  performance  indices,  determine  the  state  weighting  matrices  in  the 
integral  quadratic  performance  indices  and  hence  the  feedback  gain  matrices.”  Current 
optimal  pole  placement  approaches  are  based  on  the  assumption  that  the  state  feedback 
controllers  optimize  a linear  system  with  a single  objective.  However,  a single  integral 
performance  index  cannot  reflect  the  dynamic  interactions  between  controller  groups. 
Therefore,  a new  optimal  pole  placement  methodology  based  on  the  differential  game  theory 
for  a general  multiobjective  regulator  problem  is  proposed  in  the  second  part  of  this  research. 
By  using  multiple  integral  quadratic  performance  indices,  the  control  domain  can  be 
expanded,  thus  allowing  the  system  control  designer  the  freedom  of  implementing 
additional  criteria  to  specify  the  system’s  performance.  The  proposed  method  is  capable  of 
shifting  either  a single  pole  or  a pair  of  poles  (two  real  poles  or  a pair  of  complex  conjugate 
poles)  to  the  desired  closed-loop  location(s)  while  satisfying  additional  design 
specifications.  Multiple  pole  shifting  tasks  can  be  implemented  by  utilizing  the  shifting 
algorithm  from  a single  pole  or  a pair  of  poles  recursively.  It  is  found  there  are  several 
advantages  to  the  proposed  methodology, 
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( 1 ) .  The  proposed  methodology  can  determine  state  feedback  controllers  in  an  n-player 

differential  game  without  actually  solving  the  n-couple  algebraic  Riccati  equations 
numerically. 

(2) .  The  proposed  method  can  incorporate  a design  specification  (e.g.,  minimize  the 

envelop  bound  of  full-order  state  time  response)  as  an  optimization  criterion. 

(3) .  The  inverse  LQR  solution  belongs  to  the  solution  set  of  the  proposed  optimization 

problem. 

(4) .  The  proposed  methodology  can  incorporate  other  controller  design  considerations 

(e.g.,  constraints  on  the  elements  of  the  state  feedback  gain  matrices)  as  additional 
equality  or  inequality  constraints  for  this  optimization  problem. 

1.3  Dissertation  Outline 

This  dissertation  is  organized  as  follows.  In  Chapter  2,  an  introductory  review  of 
differential  game  theory  is  provided.  The  mathematical  model  for  a multiobjective  linear 
quadratic  regulator  is  also  introduced.  In  Chapters  3 and  4,  integrative  schemes  based  on 
homotopy  theory  are  proposed.  The  symmetric  property  of  a Riccati  matrix  in  conjunction 
with  the  Kronecker  sum  are  used  to  make  the  proposed  algorithms  more  efficient.  A robust 
algorithm,  which  guarantees  a positive  definite  solution  of  the  algebraic  Riccati  equation 
without  finding  all  the  solution,  is  presented  in  Chapter  3.  In  Chapter  4,  an  efficient 
algorithm  for  solving  coupled  algebraic  Riccati  equations  is  presented.  An  example  shows 
the  proposed  homotopy  scheme  can  determine  a solution,  which  is  not  solvable  by  the 
Newton’s  iterative  scheme. 

The  issue  of  pole  placement  and  weighing  matrices  selection  for  game  theoretic  state 
feedback  controllers  are  discussed  in  Chapters  5 and  6.  Chapter  5 presents  a pole  shifting 
scheme  which  can  be  implemented  by  iteratively  performing  a single  pole  shifting  or  a pair 
of  poles  shifting  task  is  proposed.  There  exist  infinite  solutions  to  the  game  theoretic 
controller  by  utilization  of  the  proposed  scheme.  For  both  cases,  it  can  be  shown  that  the 
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solution  to  the  LQR  controller  (inverse  optimal  problem)  belongs  to  the  solution  set  of  the 
game  theoretic  controller.  In  order  to  get  a finite  solution  set,  a “superimposed”  optimization 
criterion  is  needed.  Chapter  6 presents  some  additional  optimization  criteria  from  which  the 
control  engineers  can  choose.  These  optimization  criteria  are 

(1) .  Minimize  the  square  of  the  Frobenius  norm  of  the  reduced-order  state  feedback 

gain  matrix. 

(2) .  Minimize  the  envelop  bound  of  reduced-order  state  time  response. 

(3) .  Minimize  the  envelop  bound  of  full-order  state  time  response. 

The  three  minimization  criteria  are  chosen  individually  as  a superimposed  optimization 
criterion  along  the  prescribed  closed-loop  pole  location  to  form  the  three  optimization 
models.  A lateral  motion  model  of  an  F-4  aircraft  and  a longitudinal  motion  model  of  an 
AIRC  aircraft  are  used  as  numerical  examples  to  demonstrate  the  feasibility  of  the  proposed 
optimization  model.  In  Chapter  7,  a conclusion  of  this  research  as  well  as  suggestions  for 
future  work  is  provided. 


CHAPTER  2 

APPLICATION  OF  GAME  THEORETICAL  CONTROLLER  TO  MULTIOBJECTIVE 

LINEAR  SYSTEM 

2.1  Introduction 

For  many  large  scale  systems,  there  frequently  exist  many  design  performance  indices 
to  be  accomplished  by  system  engineers  [11],  Thus,  the  multiobjective  optimization 
problems,  which  study  a system  with  multiple  objectives,  have  become  a popular  research 
topic.  The  study  of  multiobjective  problem  in  optimal  control  theory  falls  into  the  scope  of 
differential  game  theory,  which  is  an  outgrowth  of  game  theory  and  optimal  control  theory. 

In  this  chapter,  an  introduction  to  multiobjective  optimization  problems  and  the 
differential  games  is  given.  The  mathematical  models,  based  on  the  differential  game  for 
the  multiobjective  linear  quadratic  regulators,  are  also  introduced. 

2.1.1  Historical  Background 

Game  theory  was  first  developed  to  analyze  economic  and/or  war  situations  involving 
the  multiple  objectives  of  several  individuals.  The  first  study  of  game  theory  was  initiated 
by  Nash  [12],  who  was  the  recipient  of  the  Nobel  Prize  for  Economics  in  1994.  Isaacs  [3] 
adopted  the  concepts  from  game  theory  in  conjunction  with  optimal  control  theory  and 
introduced  the  theory  of  the  differential  game  and  its  application  to  multiobjective  linear 
systems.  The  analysis  from  Isaacs’s  research  was  based  on  a dynamic  programming 
approach.  Later  on,  Berkowitz  and  Fleming  [13]  utilized  the  principles  from  calculus  of 
variations  to  treat  differential  games  with  a single  integral  objective,  which  is  the  prototype 
of  the  modern  differential  game.  At  that  time,  the  researchers  concentrated  on  a single 
integral  quadratic  performance  index  in  the  differential  game.  For  the  multiple  input  control 
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problem,  the  single  objective  differential  game  can  be  classified  into  two  categories,  namely 
zero-sum  differential  games  and  nonzero-sum  differential  games.  In  a 2-player  differential 
game,  a zero-sum  differential  game  happens  when  each  set  of  controller  groups  have  a direct 
conflict  of  interest;  one  group  of  controllers  chooses  the  most  favorable  action  to  an 
objective,  and  the  other  group  of  controller  chooses  the  most  unfavorable  action  to  the  same 
objective.  In  a 2-player  differential  game,  a nonzero-sum  differential  game  happens  when 
two  sets  of  controller  groups  do  not  have  completely  conflicting  interests.  The  classical 
linear  quadratic  regulator  for  a multiple  input  system,  which  has  a scalar  integral  quadratic 
performance  index  to  be  minimized,  can  be  categorized  as  a nonzero-sum  “cooperative” 
game.  It  is  because  all  controllers  are  in  the  same  group  to  minimize  the  scalar  objective 
criteria,  which  can  be  defined  as  the  “weighted”  sum  of  all  the  individual  objectives 
associated  with  each  controller.  Therefore,  a nonzero-sum  “cooperative”  game  theoretical 
controller  for  a multiobjective  linear  system  cannot  be  found  unless  information  about  the 
“degree”  of  cooperation  (or  conflict)  between  all  the  objectives  is  provided.  A cooperative 
optimal  strategy  gives  an  optimistic  solution  by  assuming  some  degree  of  cooperation  exists 
among  all  the  objectives  [12].  Starr  and  Ho  [4]  proposed  a nonzero-sum,  noncooperative 
differential  game  that  has  multiple  integral  quadratic  performance  indices  to  be  minimized. 
A nonzero-sum  noncooperative  game  gives  a conservative,  sometimes  pessimistic  solution 
which  is  based  on  the  assumption  that  “cooperation”  does  not  exist  among  all  the  controllers. 

The  definition  of  the  multiobjective  optimization  problems  is  stated  as  follows  [14]: 
given  an  admissible  control  space,  U,  the  behavior  of  a system  is  characterized  by  an 
n-dimensional  vector  (or  n-player  in  a differential  game),  u = [ux,  u2,  ...w„]r,  u G U,  and 
is  measured  by  an  m-dimensional  vector  objective  function  (or  m-cost  functions  in  a 
differential  game)  J(u ) = [J\(u),  J2(u),  ...Jm(u)]T,  the  components  of  J(u)  are  real  functions 
of  u.  It  is  required  to  determine  a point  (points)  u°  G U,  which  optimizes  the  values  of  the 
objective  functions  Jl(u),J2(u), ... Jm(u ) in  some  “sense.” 
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Solution  methodologies  for  the  multiobjective  optimization  problems  can  be  roughly 
classified  into  the  following  categories  [15]: 

(i)  Optimization  of  all  system  design  criteria  criteria  simultaneously. 

(ii)  Determination  of  a set  of  noninferior  points, 

(iii)  Optimization  of  a set  of  system  design  criteria  hierarchically, 

The  method  of  optimization  of  all  system  design  criteria  simultaneously  is  a 
conservative  strategy  in  the  sense  that  it  assumes  that  there  is  no  cooperative  situation  among 
all  multiple  objectives.  The  parallel  strategy  in  game  theoretic  is  called  “Nash  strategy”  or 
“Nash  equilibrium.”  For  a given  admissible  control  space,  U,  a strategy  set  u E U,  where 
u — \u\,U2,  —un\,  is  called  a Nash  strategy  set  in  U relative  to  J{u),  if  for  i = 1 

Ji(u\,U2,  ...un)  < Ji(u\,...,Ui-\,ui,Ui+  [4],  In  other  words,  a Nash  strategy  is  an 

optimal  strategy  for  each  of  the  players  based  on  the  assumption  that  all  of  the  other  players 
are  sticking  to  their  Nash  equilibrium.  In  a noncooperative  game,  where  the  cooperation 
among  the  decision  makers  is  inadmissible  or  at  least  difficult  to  enforce,  Nash  strategy  is 
a reliable  solution.  This  solution  is  secured  against  any  attempt  by  one  player  to  unilaterally 
alter  its  strategy,  since  any  player  can  only  lose  by  deviating  from  the  equilibrium  solution. 
A Nash  strategy  is  applicable  in  the  dynamical  system  with  conflicting  objectives  like 
guidance  problems  [16]  and  collision  avoidance  strategies  in  robot  mechanism  [17]. 

The  largest  collection  of  research  work  is  devoted  to  the  method  of  determination  of 
a set  of  noninferior  points.  For  a given  admissible  control  space,  U,  a point  w°  E U is  called 
a noninferior  point  in  U relative  to  J(u),  if  among  all  u°  E U,  there  does  not  exist  a point 
u such  that  Jt{u)  < i = 1,2,  ...k,  with  at  least  one  of  the  inequalities  being  strict 

[18].  The  parallel  strategy  in  game  theoretic  is  called  “Pareto  strategy”  or  “Pareto  optimum.” 
Many  algorithms  have  been  developed  to  implement  the  Pareto  strategy  in  multiobjective 
dynamical  systems  [11,19,20,21], 
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The  general  procedure  to  find  a Pareto  optimum  can  be  implemented  in  two  steps  [11]. 
The  first  step  is  to  determine  the  degree  of  compromise  between  the  conflicting  objectives 
in  some  numerical  sense.  For  example,  in  the  two  objective  optimization  problem,  if  the  two 
objectives  are  “equally”  important,  then  two  weights,  a,  =0.5  and  a2  = 0 . 5,  will  be 
assigned  to  the  associated  objectives.  Once  an  “acceptable”  compromise  is  reached,  the 
multiobjective  optimization  problem  can  be  converted  into  a scalar  (or  single)  objective 
optimization  by  summing  all  the  multiplication  of  performance  indices  with  their  associated 
weights.  For  instance,  the  two  objective  optimization  example  is  converted  to  finding  the 
optimum  of  a single  objective  J,  where  J = axJx  + a2J2.  Notice  that  the  relative 
importance  of  these  performance  indices  is  generally  unknown  until  the  system’s  best 
capabilities  are  achieved  and  the  trade-offs  between  the  performance  indices  reached.  An 
important  inference  from  the  above  example  is  that  solutions  from  the  classical  single 
performance  index  regulator  problem  belong  to  the  Pareto  solution  set,  since  weights 
associated  with  each  performance  index  are  pre-chosen.  The  major  drawback  of  classical 
optimal  control  technique  to  the  regulator  problem  is  that  the  solution  is  only  optimal  to  a 
chosen  weighted  sum  of  all  performance  indices,  but  not  necessary  to  the  system.  In  other 
words,  there  may  exist  another  solution  (e.g.,  the  Pareto  optimal  solution),  with  its  associated 
weighted  sum  of  performance  indices,  which  will  result  in  a better  system  performance  than 
the  current  chosen  one.  Thus,  an  “optimal”  Pareto  solution  is  the  solution  (in  the  Pareto 
solution  set)  associated  with  the  “optimal”  weighting  coefficient  set. 

By  using  a Pareto  strategy  in  a multiobjective  optimization  problem,  the  condition 
characterizing  a compromised  solution  can  be  expressed  in  an  analytic  form  such  that  the 
trade-off  alternatives  can  be  determined  with  conventional  optimization  techniques.  This 
condition,  which  follows  the  ordinary  meaning  of  “compromise,”  is  that  “no  improvement 
for  any  performance  index  can  be  achieved  except  at  the  expense  of  a degradation  in  at  least 
one  of  the  other  performance  indices”  [11], 
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The  method  of  optimization  of  system  design  criteria  hierarchically  is  based  upon  the 
introduction  of  a preference  ordering  to  the  set  of  optimization  criteria  such  that  the 
multiobjective  optimization  problem  is  implemented  by  optimizing  a set  of  scalar  criteria 
sequentially.  The  parallel  methodology  in  game  theory  is  called  the  “Stackelberg  strategy” 
or  “Stackelberg  optimum.”  The  Stackelberg  strategy  requires  the  performance  indices 
ranked  in  their  order  of  importance  [22],  The  optimal  value  of  the  primary  index  alone  is 
found  first.  The  optimal  solution  of  the  secondary  performance  index  is  then  subject  to  the 
constraint  that  the  value  of  the  primary  index  is  held  within  some  prescribed  bonds  of  its 
optimum.  The  third  performance  index  is  treated  similarly,  with  the  first  two  optima 
included  as  additional  constraints  and  so  on,  until  all  the  performance  indices  have  been 
considered.  One  typical  example  of  using  the  Stackelberg  strategy  in  the  multiobjective 
optimization  problem  is  the  leader-follower  problem  [23],  In  a leader-follower  problem,  the 
follower’s  objectives  serve  as  the  first  performance  indices  set  to  be  optimized,  since  the 
leaders  must  make  their  move  according  to  followers’  decision,  which  is  optimal  to  the 
followers. 

Another  important  game  theoretic  strategy  that  falls  into  this  category  is  the  “Mini-Max 
strategy.”  The  major  difference  between  the  Mini-Max  strategy  and  the  Stackelberg  strategy 
is  that  the  former  one  has  only  one  performance  index  and  the  latter  one  has  multiple 
performance  indices  in  the  differential  games.  For  a Mini-Max  strategy,  when  Player  1 
believes  that  the  other  players  play  a Nash  strategy,  Player  1 should  also  play  a Nash  strategy 
to  avoid  deviating  from  his  minimum.  However,  if  Player  1 cannot  be  sure  of  how  his  rivals 
select  their  strategies,  Player  1 may  choose  to  minimize  his  cost  against  the  worst  possible 
set  of  strategies  his  opponents  could  choose  [17].  Therefore,  it  is  an  excessively  pessimistic 
strategy,  since  Player  l’s  rivals  may  also  choose  their  Nash  strategies.  In  a 2-player 
Mini-Max  game,  there  is  usually  only  one  performance  index  for  the  two  players,  a 
minimizer  and  a maximizer.  The  optimal  value  for  the  maximizer  is  determined  first.  The 
resulting  optimum  to  the  maximizer  will  be  treated  as  an  additional  constraint  while  the 
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minimizer  finds  his  optimum.  One  application  of  Mini-Max  strategy  in  a multiobjective 
optimal  control  example  is  that  of  a Mini-Max  or  predator-prey  problem  [17].  The  predator, 
who  wants  to  minimize  the  distance  between  predator  and  prey,  serves  as  a minimizer.  On 
the  contrary,  the  prey,  who  wants  to  keep  away  from  the  predator  as  far  as  possible,  serves 
as  a maximizer. 

However,  the  problem  with  the  application  of  a Stackelberg  strategy  stems  from  the  fact 
that  sometimes  it  is  very  difficult  to  determine  the  order  of  importance  for  all  optimization 
criteria.  Therefore,  the  application  of  a Stackelberg  strategy  becomes  less  effective  for 
practical  problems  because  optimization  with  respect  to  the  chosen  “most”  important 
criterion  has  already  led  to  a unique  optimal  solution  and  optimization  of  the  other 
performance  indices  is  only  “sub-optimal”  with  respect  to  the  first  performance  index.  A 
good  example  is  the  predator-prey  problem.  If  the  predator’s  performance  index  is  chosen 
as  the  first  objective  to  be  optimized,  then  the  optimization  result  is  always  in  favor  of  the 
predator,  and  vice  versa.  In  general,  the  Stackelberg  optimum  may  not  produce  acceptable 
results.  For  example,  the  prey  may  not  want  to  follow  the  predator’s  optimum  because  the 
optimum  is  favorable  to  the  opponent. 

2.1.2  Comparison  among  Stackelberg.  Pareto  and  Nash  Strategies 

An  example  of  2-player  nonzero  sum  static/dynamic  game  [14,24,25]  is  shown  in  Fig. 
2.1,  where  uxu2  E R,  are  feasible  strategies  for  Player  1 (wj)  and  Player  2 (u2),  and  the  cost 
functions  71(w1,  w2)  and  J2(ux,u2)  are  convex  and  twice  differentiable  with  respect  to  both 
arguments.  Player  1 wants  to  minimize  Jx  while  Player  2 minimizes  J2.  Equal  cost  contours 
in  the  space  expanded  by  Ux  and  U2  are  plotted  for  7,  and  J2  in  Fig.  2.1.  Now,  suppose 
Player  2 plays  the  game  first  by  choosing  w2t-  Player  1 will  choose  ulx  such  that 
Jx(ulx,u2x)  < Jx(ux,u2x),  where  ux  E Ux.  Player  1 can  find  by  drawing  a horizontal 
line,  u.  = u2x,  which  is  tangent  to  an  equal  cost  contour  7,  of  Player  1.  The  f/j-coordinate 
of  the  tangent  point  is  ux  x,  which  minimize  the  cost  7,  when  u2x  is  applied.  Similarly,  for 


13 


Figure  2.1  “AT’  is  the  location  of  the  Nash  strategy  solution,  “S;2”  is  the  location  of  the 
Stackelberg  strategy  solution  with  Player  1 as  a leader,  “ S2] ” is  the  location  of 
the  Stackelberg  strategy  solution  with  Player  2 as  a leader,  and  the  reaction 
curve,  0,02,  contains  location  of  all  possible  Pareto  strategy  solutions. 
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the  other  choice  of  Player  2’s  strategy,  Player  1 can  find  a “minimum”  point.  By  repeating 
this  procedure  over  the  entire  Ux  x U2  space,  a reaction  curve  of  Player  1,  Px  Qx,  can  be 
found.  Correspondingly,  if  Player  1 plays  first,  then  a reaction  curve  of  Player  2,  P2Q 2,  can 
be  found. 

In  a Stackelberg  game  with  Player  2 serving  as  a leader,  Player  1 will  make  a move 
according  to  Player  l’s  reaction  curve,  PXQX.  Hence,  Player  2 will  choose  his  strategy  by 
finding  the  minimum  of  J2  along  the  reaction  curve  PXQX.  This  can  be  achieved  by  finding 
a equal  cost  contour  of  J2  which  is  tangent  to  PXQX.  Such  a tangent  point,  S21,  as  shown 
in  Fig.  2.1,  is  a Stackelberg  optimum  with  Player  2 serving  as  the  leader.  Likewise,  SX2  in 
Fig.  2.1  is  the  Stackelberg  solution  with  Player  1 being  the  leader. 

In  a Nash  game,  the  strategy  pair  (ux^  w2^)  has  the  following  properties  that 

jx(uXn,u2n)  < Jx(ux,u2N)and  J2(uXn,u2n)  < J2{uXn,  u2)  for  any  (ux,u2)  in  the  Ux  X U2 

space.  The  intersection  point,  N,  of  the  reaction  curves  PXQX  and  P2Q2  gives  the  Nash 
equilibrium  solution  for  the  game. 

The  cost  for  the  leader  in  a Stackelberg  game  is  the  best  he  can  have  for  any  pair  of 
strategies  on  the  reaction  curve  of  the  follower.  So  the  cost  for  the  leader  is  certainly  at  least 
as  good  as  his  cost  of  when  a Nash  strategy  is  used,  provided  that  the  follower  plays  along 
follower’s  reaction  curve.  In  Fig.  2.1,  both  Stackelberg  solutions  are  better  than  the  Nash 
solution  for  both  players.  Situations  where  the  follower  will  be  worse  off  than  when  a 
Stackelberg  strategy  is  played  in  a nonzero  sum  game  also  arise.  However,  if  the  leader  does 
choose  a strategy  corresponding  to  a Stackelberg  strategy,  the  follower  will  do  worse  by  not 
following  a Stackelberg  solution  himself. 

In  a Pareto  game,  the  “scalar”  objective  function  can  be  found  by  taking  the  weighted 
sum  of  all  the  objective  functions.  The  Pareto  solutions  form  the  curve  OxO~,  by  connecting 
all  the  tangent  points  of  equal  cost  functions  Jx  and  equal  cost  functions  . Again,  it  is  clear 
to  see  that  classical  optimal  control  is  only  one  point  on  the  0X02  curve,  since  the  single 
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performance  index  in  the  classical  optimal  control  problem  is  the  weighted  sum  of  all  the 
performance  indices. 

A Nash  game  is  also  entitled  to  a noncooperative  game,  since  there  is  no  cooperation 
between  players.  Similarly,  a Pareto  game  is  also  called  a cooperative  game  since  “all” 
players  agree  to  cooperate.  Examples  of  the  comparison  between  cooperative  and 
noncooperative  infinite  time  horizon  differential  games  are  made  in  References  4 and  26. 
It  is  often  difficult  to  classify  a game  as  either  noncooperative  or  cooperative.  In  reality, 
the  cooperation  situation  may  change  from  time  to  time  in  the  differential  game  [27],  If  all 
the  players  benefit  from  cooperation  in  a time  interval,  then  the  Pareto  solution  is  the  optimal 
solution  in  the  duration.  However,  if  there  exists  one  player  who  will  not  benefit  from  the 
cooperation,  then  the  Nash  solution  is  the  reasonable  solution  in  the  time  interval. 

2.2  Multiobjective  Linear  Quadratic  Regulator 

For  conciseness,  the  following  historical  review  is  based  on  a 2-objective  2-player 
differential  game.  Extension  to  higher  order  differential  games  follows  directly.  A linear 
dynamical  system  which  mathematically  models  a controlled  and  observed  system  can  be 
represented  in  the  state  space  form, 

x = Ax(t)  + Bxux(t)  + B2u2(t ) (2.1) 

y(t)  = Cx(t ) 

where  ux(t)  E 9kmi  and  u2(t)  E 9km2  represent  separate  control  inputs  to  the  plant  (Player 
1 and  Player  2 of  the  game,  respectively),  and  y(t ) E 3k*  represents  the  plant  outputs; 
x(t)  E 9k”  represents  the  plant  states  (current  condition  of  the  game)  and  is  assumed 
perfectly  measured.  The  control  actions  ux  and  u2  are  restricted  to  state  feedback  (i.e., 
ui  = M((x(0)>  i = 1,2).  Matrices  A,  Bx,  B2,  and  C are  appropriately  dimensioned,  and 
time-invariant. 
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Application  of  game  theoretic  controllers  with  Nash,  Pareto,  Mini-Max  and 
Stackelberg  strategies  to  the  multiobjective  Linear  Quadratic  Regulator  problems  is 
discussed  in  the  following  sub-sections. 

2.2.1  Game  Theoretic  Controller  with  Nash  Strategy 

A Nash  game  or  noncooperative  nonzero-sum  differential  game,  is  formulated  by 
allowing  the  controllers  (players)  to  separately  minimize  integral  quadratic  performance 
indices,  J { and  J2,  subject  to  the  dynamical  constraints  imposed  by  Eq.  (2.1).  The  solution 
is  based  on  the  assumption  that  the  players  know  each  other’s  performance  indices  and  when 
the  strategies  have  been  calculated,  they  are  announced  at  that  time.  Considering  the  infinite 
time  horizon  case,  we  define  these  integral  quadratic  performance  indices  as 

00 

./|(Wj,W2)  ^ (x  2jX  T WjRjjWj  "t-  u2RX2u2)  dt 

o 

00 

■^2^1’ ^2 ) i 2 2*  Wj/?2]wi  T 1^2^ 22ut)  (2.2) 

0 

where  2j,  Q2,  Rn,  and  R2l  are  positive  semi-definite  matrices,  and  Rn,  R 22  are  strictly 
positive  definite  matrices  [28], 

The  Nash  control  actions  u\  and  u\  are  extremals  of  Eq.  (2.2)  and  are  subjected  to  the 
dynamical  constraint  equation  (Eq.  (2.1)),  if  they  satisfy  the  following  conditions: 

J\(u\,u*2. ) < Jx(ux,u*2) 

J2(u\,u2)  < J2(u\,  u2)  (2.3) 

The  conditions  in  Eq.  (2.3)  are  known  as  the  Nash  equilibrium  conditions  [1,4,6,28], 
It  can  be  shown  that  for  these  conditions  to  exist  [28],  the  impulse  response  functions, 
dHjdu^i  = 1,2,  and  the  influence  functions,  A(-,  i = 1, 2,  must  satisfy  the  following  set 
of  modified  Euler-Lagrange  equations 
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= 0 


(2.4) 


where  Hi  is  the  Hamiltonian  associated  with  performance  index  and  is  give  by 


The  summation  terms  in  Eq.  (2.5)  reflect  the  major  difference  between  a 
noncooperative  nonzero-sum  differential  game  and  any  other  differential  games  (e.g., 
noncooperative  zero-sum  games  and  cooperative  games).  These  terms  represent  the  effects 
changes  in  control  strategy  Uj  would  have  on  control  strategies  u-,  that  is,  the  effects  of 
controller  cross-coupling.  If  each  controller  uses  his  state  feedback  strategy,  then  any 
perturbation  due  to  one  player  will  result  in  changes  of  all  the  co-state  variables  A,-  (through 
Eq.  (2.5))  and  hence  changes  of  the  state  variables  jc.  Consequently,  all  the  controllers  will 
be  immediately  influenced  by  the  resulting  perturbation  of  the  state  variables,  since  all  the 
controller  uses  state  feedback  strategies.  The  explicit  way  of  information  exchange  among 
all  the  controllers  makes  the  noncooperative  nonzero-sum  differential  game  different  from 
any  other  differential  game,  which  uses  an  implicit  pattern  of  information  exchange  (i.e.,  the 
information  exchanges  among  all  the  state  feedback  controllers  through  their  single 
performance  index).  This  is  the  reason  why  the  noncooperative  nonzero-sum  differential 
game  is  uniquely  suited  for  problems  involving  conflicting  situations  among  players 
(controllers). 

If  linear  full-state  feedback  is  assumed,  the  control  strategies  u\  and  u\  satisfying  Eq. 
(2.3)  are  [28] 


Hi  = xTQpc  + u]Rnul  + u]Ri2u2  + Xf(Ax  + Blul  + B2u2 ) (2.6) 


u 


(2.7) 


where  Pif  i — 1 , 2 are  solutions  of  the  coupled  algebraic  Riccati  equations 
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- PXA  - ATPX  - Qx  + PlBlR~lBTlPl  + PxB2R~^Bt2P2 

+ P2B2R22'Bt2Px  — P2B2R22^RX2R22B2P 2 — 0 (2.8) 

-PjA-  atp2  -q2  + p2b2r22b2p 2 + p2bxr^b\px 

+ PxBxR^B\P2  - PXBXR^R2XR^B\PX  = 0 (2.9) 

2.2.2  Game  Theoretic  Controller  with  Pareto  Strategy 

For  a 2-player  game,  the  Pareto  game  or  cooperative  differential  game  is  formulated 
by  allowing  joint  strategies  to  minimize  Jc  and  subject  it  to  the  dynamical  constraints 
imposed  by  Eq.  (2.1),  such  that 

Jx(u*i,u*2 ) < Jx(ux,u2 ) 

J2{u[,u*2)  < J2(ux,u2)  (2.10) 

where  at  least  one  of  the  inequalities  is  strict  [18],  Consider  the  infinite  time  horizon  case, 
the  performance  indices  are  defined  as 

00 

J\(ux,u2)  J (x  Qxx  -f-  uxRxxiix  -f  u2RX2u2)  dt 
o 

00 

J2(M\,  u2^  f (■%  Q2x  ~b  uxR2Xux  "f  u2R22u2)  dt  (2.11) 

0 

where  Qx,  Q2,  Rn,  and  R1X  are  positive  semi-definite  matrices  and  /?H,  R22  are  strictly 
positive  definite  matrices.  For  a convex  problem  [11],  the  Pareto  solution  can  be  obtained 
from 

00 

h = B\J\  + p2J2  = j ( xtQcX  + utRcu ) dt  (2.12) 

o 


where 
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uT  = [u\  uT,] 


(2.13) 


Qc  ~ M\Q\  3"  H2Q2 


(2.14) 


H\R\\  + P^2\ 
0 


0 

^1^12  + ^2^22 


(2.15) 


and  the  scalar  weight  coefficients  p x and  p2  satisfy 


fX  ] + fx2  ~ 1 


(2.16) 


where  /ux  > 0 and  p2  > 0. 

As  shown  in  Fig.  2.1,  the  Pareto  solution  is  not  unique;  different  values  for  pl  lead  to 
different  solutions.  One  way  to  determine  the  weight  coefficients  is  based  on  the  importance 
between  two  performance  indices  [6],  If  Jx  is  more  important  than  79,  then  p x is  assigned 
a larger  value  (i.e.,  jux  >0.5)  than  ju2,  and  vice  versa.  If  both  performance  indices  are 
equally  important,  then  both  weighting  coefficients  are  assigned  values  of  0.5.  After 
determining  the  weighting  coefficients,  the  classical  optimal  control  theory  can  be  directly 
applied  to  find  a minimum  feedback  solution. 

However,  it  is  worthwhile  to  reiterate  that  the  relative  importance  of  these  performance 
indices  is  not  generally  known,  until  the  system’s  best  capabilities  are  achieved  and  the 
trade-offs  between  the  performance  indices  reached.  This  is  the  major  drawback  for  the 
application  of  classical  optimal  control  theory  to  linear  regulator  problems,  since  those 
weighting  coefficients  are  not  easily  determined  such  that  the  best  system  performance  can 
be  achieved. 

2.2.3  Mini-Max  Strategy 

The  Mini-Max  game  or  noncooperative  zero-sum  differential  game,  is  formulated  by 
allowing  the  minimizer  (maximizer)  to  minimize  (maximize)  a performance  index  (7)  first, 
then  the  maximizer  (minimizer)  to  maximize  (minimize)  the  same  performance  index  (7) 
subject  to  the  dynamical  constraints  imposed  by  Eq.  (2.1)  and  the  optimal  strategy  of 
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minimizer  (maximizer).  Considering  the  infinite  time  horizon  case,  we  define  a performance 
index  as 


J(^UX,U2)  J ( X Qx  T U2R22U2)  ^ (2.17) 

0 

where  Q is  a positive  semi-definite  matrix,  and  Rn,  R22  are  strictly  positive  definite 
matrices.  When  constructing  the  above  performance  index,  we  have  assumed  that  ux  is  the 
minimizer  and  u2  is  the  maximizer. 

The  Mini-Max  control  actions  u*  and  u*2  are  extremals  of  Eq.  (2.17),  subjects  of  the 
dynamical  constraint  of  Eq.  (2.1),  if  they  satisfy  following  conditions: 

J(u*hu2)  < J(u\,u2)  < J(u*,u2)  (2.18) 

The  conditions  Eq.  (2.18)  are  known  as  the  Mini-Max  equilibrium  conditions.  It  has  been 
shown  that  for  these  conditions  to  exist  [17],  the  impulse  response  functions,  dH/du and 
the  influence  functions,  A,  must  satisfy  the  following  set  of  Euler-Lagrange  equations 


dH 

dUi 


U 2,  U,\ 


= 0 


^ = 0 (2-19) 

and  H is  the  Hamiltonian  associated  with  the  performance  index  J that  is  given  by 

H = xrQx  + u\Rnux  - u2R22u2  + At{Ax  + Bxux  + B2u2)  (2.20) 

If  linear  state  feedback  is  assumed,  the  control  strategies  u\  and  u2  satisfying  Eqs.  (2.1) 
and  (2. 1 8)  are 

«i  = - RXXB\A  (2.21) 

u*2  = + R22xBt2A  (2.22) 

Now,  assuming  maximizer  (u2)  moves  first,  the  optimal  control  problem  becomes  that  of 
determining  the  minimum  of  the  performance  index 
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J(u\,u2)  = J (xtQx  + u\Ruux  - ut2R21u2)  dt  (2.23) 

o 

subject  to  the  dynamic  constraints 

x = Ax(t)  + Bxux(t)  + B2u2(t ) 
u\  = R22lB^X 

y(t ) = Cx(t)  (2.24) 

where  the  second  constraint  comes  from  maximizer’s  strategy.  The  feedback  solutions  of 
the  one-side  optimal  control  problem  are 

«1  = - R^B\Px  (2.25) 

u*2  = + R22B2Px  (2.26) 

where  P is  the  solution  of  the  algebraic  Riccati  equation 

- PA  - AtP  - Q + PBXR~XXB\P  - PB2R22xBt2P  = 0 (2.27) 

Relationship  to  //°°  Optimal  Control  Problems 

A differential  game  with  a Mini-Max  strategy  for  optimal  disturbance  attenuation  can 
be  equivalent  to  a Hx  optimal  control  problem,  if  one  set  of  controller  is  a square 
integratable  disturbance  that  depends  on  current  state  [29].  For  such  a case,  the  dynamical 
system  is 

x = Ax  + Bxu  + B2w,  jc(0)  = 0 (2.28) 

For  the  infinite  time  horizon  case,  a linear  quadratic  cost  function  can  be  defined  as 

00 

J = j (■ xTQx  + utRuu)  dt  (2.29) 

o 
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where  Q > 0,  R > 0.  Let  Tu  be  the  mapping  from  w to  z = (x,  u).  The  disturbance 
attenuation  problem  is  defined  as  follows.  Find  a state  feedback  controller  to  minimize 
< Tu> 

ltf  < Tu>=  y*  > 0 (2.30) 

where 

< > = ? II  T.  | / ||  w || 

A °° 

Notice  that  ||rM||2  = J and  ||w||2=  j wTR22w  dt.  The  optimal  state  feedback 

o 

solution  and  maximum  disturbance  consists  of  the  saddle  point  of  the  game  with  kernel 

JY  =11  Tu  II2  - y2  ||  W II2  = J - y2  ||  w II2,  (2.31) 

where  the  saddle  point  (u  ,w  ) statisfies  the  following  inequities 

Jy(u*,w ) < Jy(u*,w*)  < JY(u,w*)  (2.32) 

The  optimal  state  feedback  control  is 

u*  = - R~lB]Px  (2.33) 

and  its  associated  maximum  disturbance  is 

= jiR22B2Px  (2.34) 

Matrix  P in  Eqs.  (2.33)  and  (2.34)  is  the  smallest  nonnegative  definite  solution  of  the 
following  equation 

ATP  + PA  + Q - P{BXR~XXB]  - \B2R22BT2)P  = 0 (2.35) 

2.2.4  Stackelberg  Strategy 

In  a 2-player  game,  if  one  player  has  to  announce  his  strategy  first  due  to  some  bias  in 
the  information  situation  (i.e.,  this  player  does  not  have  his  opponent’s  information; 
however,  his  opponent  knows  of  his  performance  index),  this  player  should  choose  a 
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Stackelberg  solution  for  himself  [25],  A Stackelberg  solution  for  a 2-player  nonzero-sum 
differential  game  with  biased  information  patterns  is  stated  as  follow.  Given  a 2-player 
game,  Player  1 and  Player  2 want  to  minimize  their  performance  indices 


J j(m  j,  M2) 


J 2(^1  > u2) 


00 

I 


( XTQjX  + Mj/JjjMj  + U^R^U 2)  dt 


0 


j (XTQ2X  + + u2R22u2)  dt 

0 


(2.36) 


respectively.  The  strategy  set  (u\,  u2)  is  called  a Stackelberg  strategy  with  Player  2 as  leader 
and  Player  1 as  follower  if 

J2(u*hu2)  < J2(u°(u2),  u2)  (2.37) 

where 


J\(u°(u2),  u2)  = min  Jx(ux,u2)  (2.38) 

“1 

u\  = u°(u2)  (2.39) 

As  already  mentioned  in  Section  2. 1 , in  a Stackelberg  game,  the  leader  (Player  2 in  this 
case)  will  announce  his  strategy  according  to  the  follower’s  reaction  strategy.  Hence,  the 
follower’s  performance  index  is  the  first  objective  to  be  minimized  in  order  to  determine  the 
leader  s strategy.  The  impulse  response  functions,  dH^/du^,  and  the  influence  functions, 
Aj,  for  Player  1 must  satisfy  the  following  set  of  Euler-Lagrange  equations 


dul 


= 0 


(2.40) 


Qxx  - AtXx 


(2.41) 


where  H x is  the  Hamiltonian  associated  with  performance  index  J x that  is  given  by 
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H\  = \(xTQi*  + u]Rnu\  + u[Rnu2)  + X[{Ax  + Bxux  + B2u2 ) (2.42) 

Equations  (2.40)  and  (2.41)  are  the  follower’s  reaction  strategy  which  the  leader  must 
choose.  Hence,  in  the  next  optimization  step,  Eqs.  (2.40)  and  (2.41)  can  be  treated  as 
additional  constrains  imposed  to  the  second  optimization  problem,  and  should  be  included 
in  Hamiltonian  H2.  The  impulse  response  functions,  dH2/ du2,  and  the  influence  functions, 
X2,  for  Player  2,  must  satisfy 


dH  2 
du2 

= 0 

(2.43) 

«2 

T 

dH2 

x2  = 

dx 

(2.44) 

T 

dH2 

a2  = 

dA, 

(2.45) 

The  Hamiltonian  associated  with  performance  index  is 

H2  = \(xTQ&  + u]R2Xux  + u[R22u2)  + Xt2{Ax  + Bxux  + B2u2 ) 


~ al(Qxx  + AtXx)  (2  46) 

The  open-loop  solutions  are  given  by 

«1  = - R\\BTXPX x (2.47) 

1^2  R 22  ^2^ 2 ^ (2.48) 

where  P x and  P 2 are  solution  of  the  following  coupled  equations 

- PXA  - AtPx  -Qx+  PxBxR~lB]Px  + PxB2R22xBt2P2  = 0 (2.49) 


- P2A  - AtP2  - Q2  + P2BxRu]B]Px  + P2B2R22'BT2p2  + QXP  = 0 (2.50) 

with  P being  the  solution  of  the  following  equation 

— pa  — atp  + pbxr~1btxpx  -i-  pb2r221bt2p2 
- b2r-%xr-'btxpx  + bxr^btxp2  = 0 


(2.51) 
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Equations  (2.49)  and  (2.50)  are  identical  to  the  open-loop  Nash  strategic  solution,  except  for 
the  additional  term  Q\P  contained  in  Eq.  (2.50).  The  extra  term  reflects  that  the  leader 
(Player  2)  minimizes  his  cost  on  the  reaction  strategy  of  the  follower  (Player  1). 

2.3  Summary 

In  this  chapter,  historical  reviews  for  both  the  multiobjective  optimization  problems 
and  game  theory  are  given.  The  differential  game,  which  is  the  study  of  the  optimal  reactions 
among  all  the  player  under  all  possible  cooperation  and/or  conflicting  situations  involved 
in  the  dynamical  game,  is  also  reviewed.  For  the  noncooperative  nonzero-sum  differential 
game,  information  is  exchanged  explicitly  among  all  the  controllers  by  utilization  of  the 
Nash  strategy.  This  explicit  information  exchange  pattern  differs  from  the  classical  optimal 
controller,  whose  information  exchange  among  all  the  state  feedback  controller  is  through 
its  single  performance  index.  For  the  multiobjective  linear  quadratic  problem,  the  Nash 
strategy  is  the  only  controllers  that  can  explicitly  exchange  dynamic  information;  for  any 
other  strategy  (i.e.,  Pareto  strategy,  Stackelberg  strategy  and  Mini-Max  strategy),  controllers 
can  only  exchange  information  implicitly.  This  is  the  reason  why  the  noncooperative 
nonzero-sum  differential  game  is  uniquely  suited  for  multiobjective  linear  quadratic 
problems  involving  conflicting  situations  among  players  (controllers).  Thus,  it  motivates 
this  research  to  design  game  theoretic  controllers  using  Nash  strategies. 


CHAPTER  3 

A ROBUST  HOMOTOPY  ALGORITHM  FOR 
THE  ALGEBRAIC  RICCATI  EQUATION 

3.1  Introduction 

State  feedback  controllers  have  been  successfully  developed  and  used  to  alter  the 
system  response  of  multiple  input/multiple  output  linear  dynamical  systems.  Optimal  state 
feedback  inputs  can  be  generated  by  utilizing  the  Linear  Quadratic  Regulation  (LQR) 
technique  [31,32],  The  time  varying  state  feedback  gain  is  determined  via  the  solution  of 
a differential  Riccati  equation.  In  the  infinite  time  horizon  case,  the  state  feedback  system 
is  stable  if  the  constant  solution  of  the  algebraic  Riccati  equation  is  positive  definite  [17], 
Solution  methodologies  have  been  intensively  investigated  in  the  past  two  decades 
[33,34,35],  However,  these  solution  techniques  can  only  be  applied  to  the  “standard” 
algebraic  Riccati  equation.  For  example,  the  “modified”  version  of  the  algebraic  Riccati 
equation  (i.e.,  resulted  from  the  system’s  uncertainty)  [36]  or  the  “coupled”  algebraic  Riccati 
equation  [1,6]  (or  see  Chapters  2 and  4 for  detail)  cannot  be  directly  solved  by  any 
conventional  algebraic  Riccati  equation  solver.  The  “modified”  version  of  the  algebraic 
Riccati  equation  has  the  form 

AtP  + PA  - PSP  + Q + F(P)  = 0 
In  optimal  control  of  reduced  large-dimensional  systems  [37], 

F(P)  = (/  - r )PSP(I  - t7) 

with  r as  the  non-symmetric  idempotent.  In  the  frequency-dependent  uncertainty  problem 
[38], 

F(P)  = 2 A,PA, 
i=\ 
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with  A i = block  diag  (0, 


0 


- ( o 


0) 


,0). 


In  the  decentralized  H°°  optimal  control 


problem  [39], 


F(P)  = (P  - £ TjPr,-)  X GOP  ~ | 
f=i  «=i 

k 

with  X rf  = /„. 

i=  l 

Two  categories  of  numerical  methods  have  been  used  to  solve  algebraic  Riccati-type 
equations,  namely,  iterative  methods  and  integrative  methods.  In  general,  an  iterative 
algorithm  takes  less  computational  operating  numbers  than  an  integrative  method. 
However,  for  solving  the  “modified”  or  “coupled”  algebraic  Riccati  equation,  iterative 
schemes  may  diverge  due  to  bad  initial  guesses.  For  these  types  of  Riccati  equations,  an 
integrative  algorithm  based  on  homotopy  theory  (even  thought  it  may  be  less 
computationally  efficient)  are  the  more  robust  solvers.  Thus  a homotopy  algorithm  is 
proposed  to  solve  the  algebraic  Riccati  equation  in  this  chapter.  The  algorithm  developed 
in  this  chapter  can  be  extended  to  solve  the  “modified”  or  “coupled”  algebraic  Riccati 
equations.  The  extension  to  coupled  algebraic  Riccati  equations  is  presented  in  Chapter  4. 

Homotopy  methods  have  been  proven  and  implemented  as  globally  convergent 
algorithms  to  numerically  determine  the  solutions  of  a system  of  polynomial  equations  [8], 
Watson  [10]  developed  some  generic  codes,  based  on  the  homotopy  theory,  which  can  locate 
all  the  zeros  of  a system  of  polynomial  equations.  All  the  “modified”  or  “coupled”  algebraic 
Riccati-type  equations  can  be  solved  by  homotopy  methods,  since  they  all  can  be  expressed 
in  the  form  of  a system  of  polynomial  equations.  However,  too  much  computational  effort 
is  required  to  determine  all  the  solution  of  the  “matrix”  algebraic  Riccati  equation.  For  an 
n-by-n  system,  there  exists  n2/l  possible  solutions  for  the  algebraic  Riccati  equation,  even 
though  the  unique  positive  definite  solution  is  the  only  solution  of  interest  for  the  class  of 
problems  being  examined.  Jamshidi  and  Bottiger  [40]  first  proposed  both  Q-imbedding  and 
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S-imbedding  algorithms  to  find  a solution  of  the  algebraic  Riccati  equation.  Wang  and  Pan 
[9]  generated  a homotopy  algorithm  to  find  a solution  of  the  algebraic  Riccati  equation. 
Richter,  Hodel  and  Pruett  [36]  used  a homotopy  method  for  the  numerical  solution  of  the 
modified  algebraic  Riccati  equation.  Collins,  Lawrence  and  Richter  [41]  proposed  a 
homotopy  algorithm,  which  involves  solving  two  Riccati  equations  coupled  to  two 
Lyapunov  equations,  for  maximum  entropy  design.  However,  solutions  for  the  algebraic 
Riccati-type  equation  obtained  from  either  of  the  above  algorithms  may  not  be  the  positive 
definite  solution,  which  is  the  only  solution  of  interests  in  the  optimal  control  problem. 

In  this  chapter,  a “robust”  homotopy  algorithm  which  determines  the  “positive  definite” 
solution  of  an  algebraic  Riccati  equation  is  proposed  [42].  The  algorithm  is  robust  in  the 
sense  that  the  numerical  integration  will  never  bifurcate  and  the  resulting  solution  is 
guaranteed  to  be  positive  definite.  The  remainder  of  this  chapter  is  organized  as  follows. 
In  Section  3.2,  a preliminary  introduction  to  homotopy  theory  is  given.  In  Section  3.3,  a 
robust  homotopy  algorithm  for  solving  algebraic  Riccati  equations  is  presented.  Several 
numerical  examples  and  the  summary  are  provided  in  Sections  3.4  and  3.5,  respectively. 

3.2  Preliminary  Introduction  to  the  Homotopy  Theory 

The  homotopy  theory  introduced  in  this  section  is  mainly  based  on  the  research  work 
of  Zangwill  and  Garcia  [8],  Given  a system  of  polynomial  equations  (linear  or  non-linear) 
to  be  solved,  the  homotopy  algorithm  starts  with  a simple  solution  (a  solution  to  a similar 
but  easily  solved  system  of  polynomial  equations)  and  through  a series  of  integrations, 
reaches  the  exact  solution  of  the  original  system  of  polynomial  equations. 

Mathematical  Model: 

Let  &n  be  an  Euclidean  n space.  Given  /(x)  : %n  -*  9V1  (i.e.,  x e $>"),  find  the 
solution,  x*  = [xj,  ...  , x*n]T,  of 


m = 0 


(3.1) 
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3.2.1  Homotopy  Scheme 

Given  a system  of  polynomial  equations  (Eq.  (3.1))  to  be  solved,  the  homotopy  scheme 
proceeds  as  follow: 

(1).  Identify  an  easily  solved  system  of  polynomial  equations,  say 


e(x ) = 0 

(3.2) 

where  e(x)  : Sfe/1  -*•  Determine  the  solution  x°of  Eq.  (3.2). 

(2). 

Define  a homotopy  function  h(x,  t ),  such  that 

h(x,  0)  = e(x) 

(3.3) 

h(x,  1)  = f(x) 

(3.4) 

(3). 

Generate  a path  (from  t = 0 to  t = 1)  which  leads  the  solution  from 

0 * 
X ^ to  X . 

3.2.2  Mathematical  Models  of  Homotopv  Functions.  h(x.  t) 

Although  there  exist  numerous  forms  of  homotopy  functions,  the  following  are  the 


most  common  three: 

(i).  Newton  homotopy 

h(x,t)  = f{x)  - (1  - t)f(x°) 

(3.5) 

(ii).  Fix-point  homotopy 

h(x,  t)  = (1  - t)(x  - x°)  + tf(x) 

(3.6) 

(iii).  Linear  homotopy 

h(x,t)  = x)  + (1  - t)e(x) 

= e(x)  + t(f(x)  - e{x))  = 0 

(3.7) 

Homotopy  paths  start  from  any  arbitrary  point,  ;t0,  for  either  the  Newton  homotopy 
function  or  the  fix-point  homotopy  function;  hence,  they  are  the  most  popular.  However, 
if  one  can  find  an  easy  solved  system  of  polynomial  equations,  and  it  is  very  “close”  to  the 
original  system  of  polynomial  equations,  a linear  homotopy  function  might  be  a better 
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mathematical  model  to  work  with.  For  the  latter  case,  intuitively,  one  can  take  larger 
integration  steps,  thus  requiring  less  computational  effort.  It  should  be  noted  that  all  the 
above  homotopy  expressions  are  similar.  For  examples,  f(x)  - f(x°)  in  Eq.  (3.5)  and 

x — in  Eq.  (3.6)  are  equivalent  to  e(x)  in  Eq.  (3.7).  The  following  derivations  will  be 
based  on  the  linear  homotopy  model  only,  since  we  can  find  an  easily  solved  system  of 
polynomial  equations  (i.e.,  a Lyapunov  equation),  and  it  is  very  close  to  the  Riccati-type 
equations. 


3-2.3  Path  Following  Algorithm  and  Homotopy  Differential  Equation 


The  differentiation  of  any  homotopy  function  with  respect  to  the  homotopy  variable, 
t,  is  zero,  since  a homotopy  function  is  defined  to  be  zero  in  (x,  t),  i.e., 


dh  _ dfi  , dh  dx 
dt  dt  dx  dt 


(3.8) 


which  yields 


,dhs  - 1 dh_ 
Kdx ’ dt ‘ 


(3.9) 


Equation  (3.9)  is  referred  to  as  the  “homotopy  differential  equation”  or  “Davidenko 
differential  equation”  [8],  Integration  of  Eq.  (3.9)  over  the  “homotopy  variable”  t,  from  0 
to  1,  will  give  a solution  for  the  system  of  polynomial  equations  (Eq.  (3.1)).  Let 
x G D C %n,  and  r={H0<r<l}.  In  order  to  guarantee  the  path  existence,  the 

regularity  of  H or  the  inversion  of  the  Jacobian  matrix,  in  Eq.  (3.9)  should  always  hold 

for  all  (*,  t ) in  H~  \ where  H~ 1 is  defined  as  the  set  of  all  the  solutions  of  h(x,  t),  i.e., 

H~l  = {(x,t)  £ D x T I h(x,t ) = 0}.  (3.10) 


The  existence  of  homotopy  path  which  results  in  the  desired  solution  of  a system  of 
polynomial  equations  is  guaranteed  by  Sard’s  Theorem  (i.e.,  the  homotopy  algorithm  is 
globally  convergent  with  “probability  one”  [10]).  Thus,  the  homotopy  algorithm  can  only 
fail  for  points  which  are  in  a set  of  “measure  zero.”  Notice  that  a set  has  measure  zero  in  9&n 
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if  it  has  no  volume  in  9&n.  For  example,  a line  segment  has  a positive  measure  in  9&1,  but 
not  in  9&n  for  n > 2,  hence  a line  segment  has  measure  zero  in  9k  n for  n > 2. 

Theorem  3.2.3  Sard’s  Theorem  (Zangwill  [8],  pp.  422) 

Let  h : % —>d  9k9  — > 9k",  where  9G  is  the  closure  of  an  open  set  and  h be  C*.  If 
k — 1 + max  {0,  q — n},  then  h is  regular  for  almost  all  e,  where 

K')  = £ □ 

Corollary  3.2.3  Extended  Sard’s  Theorem  (Zangwill  [8],  pp.  422) 

Let  D C 9k"  be  the  closure  of  an  open  set,  h : D x T -»  9V1  be  C2,  and  / : D -*  &/1 
be  C1,  then  the  following  three  statements  are  equivalent, 

(i)  h will  be  regular  for  almost  all  e, 

(ii)  h will  be  regular  at  t for  almost  all  e, 

(iii)  /will  be  regular  for  almost  all  e.  q 

For  a homotopy  function,  q = n + 1,  if  k > 2,  then  h(x,  t)  is  regular  for  almost  all  e. 

In  other  words,  as  long  as  h(x,  t ) C C2,  Sard’s  theorem  states  that  for  an  arbitrary  e,  H(x,  t ) 
is  almost  assuredly  regular.  It  can  be  shown  that  all  the  systems  of  polynomial  equations  are 
continuous  and  hence  all  algebraic  Riccati  equations  are  continuous.  Thus,  homotopy 
functions  for  the  algebraic  Riccati  equations  belong  to  C2.  It  infers  that  even  if  the 
integration  path  bifurcates  (or,  the  Jacobian  matrix  is  singular)  in  a homotopy  algorithm,  we 
can  slightly  perturb  the  stating  point  of  the  homotopy  equation  such  that  a new  integration 
path  is  guaranteed  to  exist.  Sard’s  theorem  has  been  utilized  by  most  current  homotopy 
algorithms  [10,36,41]  to  bypass  the  bifurcation  points  along  the  homotopy  path. 

3.3  Homotopy  Algorithm  to  Solve  an  Algebraic  Riccati  Equation 
An  algebraic  Riccati  equation  is  defined  as 


PA  + AtP  - PBR~1BtP  + Q = 0 


(3.11) 
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where  A,  P,  Q G <&nXn,  B G eR>"Xm  and  /?  G 9V"Xw.  Matrices/5  and  R are  positive 
definite,  Q is  positive  semi-definite.  Before  developing  the  homotopy  algorithm  which 
solves  an  algebraic  Riccati  equation  (Eq.  (3.11)),  we  digress  for  a moment  to  discuss  the 
vector  operators  and  Kronecker  products.  These  properties  will  be  used  to  “vectorize”  the 
matrix  equation  which  results  from  the  homotopy  process. 

Prop.  3.3.1  Vector  operators  and  properties  of  Kronecker  products  (Graham  [43]) 

Given  matrices  A G <&mXn,  B G 3^rX's,  C G %nXn,  D G &/nXm  and  Y G %nXr, 
the  following  vector  operators  properties  exist. 

(i).  Kronecker  product: 


A®B 


auB  aX2B  ...  aXnB 
a2\B  a22B  — a2nB 

am\B  am2 ^ amnB 


G (^}mr^'ns 


(ii).  Kronecker  sum: 


C © D = C <g>  Im  + /„  ® D 

where  /,„  and  /„  are  m x m and  n x n the  identity  matrices,  respectively, 

(iii).  Vector  of  a matrix: 


Vec(A)  [a n a2X  ...  amX\aX2  a22  ...  am2 1 \aXn  a2n  ...  amn]T 

(iv) .  Vector  of  a matrix  sum  (distributive  property): 

Vec(A  + B)  = Vec(A ) + Vec(B) 

(v) .  Vector  of  a matrix  product: 

Vec(AYB)  = (BT  ®A)VecY 


□ 
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Returning  to  the  development  of  a homotopy  algorithm,  the  following  derivation  is 
based  on  a linear  homotopy  model,  since  an  easily  solved  system  of  polynomial  equations 
can  be  found  (i.e.,  a Lyapunov  equation).  Let 

f(P)  = PA  + AtP  - PBR~xBtP  + Q = 0 (3.12) 

and  choose  a starting  point  from  the  non-trivial  solution  of  e(P) 

e(P)  = P(A  + al)  + (A  + aI)TP  - PBR~XBTP  = 0 (3.13) 

where  a is  either  zero  or  a positive  random  number  plus  the  negation  of  smallest  non-positive 
eigenvalues  of  the  matrix  A (if  A has  any);  the  quantity  a is  chosen  to  ensure  that  ( A + al) 
is  strictly  positive  definite.  Notice  that  similar  techniques  of  shifting  eigenvalues  of  original 
system  matrix  have  been  found  in  Ref.  9 which  uses  a “different”  homotopy  function  to  solve 
an  algebraic  Riccati  equation  and  in  Ref.  41  which  solves  the  modified  algebraic  Riccati 
equations  resulting  from  the  maximum  entropy  problems.  The  reason  for  matrix  ( A + al) 
to  be  positive  definite  will  be  discussed  in  Section  3.3.1.  Based  on  Eqs.  (3.13)  and  (3.12), 
the  matrix  form  of  the  homotopy  function  can  be  rewritten  as 

h(P,t)  = e{P)  + t (f(P,t ) - e(P)) 

= P(A  + (1  - t)al)  + (A  + (1  - t)aI)TP  - PBR~XBTP  + tQ  = 0 (3.14) 

Differentiating  Eq.  (3.14)  with  respect  to  the  “homotopy  variable”  t yields 

PAS  + AjP  = - Q + 2aP  (3.15) 

where 

As  = A + (l  - t)al  - BR~lBrP 

P in  Eq.  (3.15)  can  be  solved  by  standard  Lyapunov  solvers.  However,  in  order  to  pave  the 
way  for  future  use  (i.e.,  the  solution  of  coupled  Lyapunov  equations  in  Chapter  4),  another 
solution  algorithm  utilizing  the  properties  of  Kronecker  products  is  developed  as  follow. 

Notice  that  the  propose  algorithm  utilizes  the  symmetric  property  of  P in  conjunction  with 
the  Kronecker  sum  to  improve  its  efficiency. 


34 


Using  Prop.  3.3.1,  Eq.  (3.15)  can  be  vectorized  as 

(Aj  © Aj)Vec(P)  = - Vec(Q)  + 2 aVec(P)  (3.16) 

where 

Vec(P)  = [pu  p2 1 ...  pn\\pn  P22  •••  Pn2 ' \P\n  P2n  — Pnn]r 

^c(P)  = [pn  p21  ...  pnl\pn  p22  ...  pj I pXn  p2n  ...  pnn]T 

Vec(Q)  = [qn  qn  ...  qnl\qn  q22  ...  qj I qu  q2n  ...  qnn ]T 

Comparing  Eqs.  (3.9)  and  (3.16),  one  observes  that  the  Kronecker  sum, 

Aj  © Aj 

defined  in  Eq.  (3.16)  is  actually  the  Jacobian  matrix. 

The  proposed  homotopy  scheme  thus  far  involves  the  solution  of  an  n2  system  of 
polynomial  equations  at  each  integration  step  for  P.  By  utilizing  the  symmetric  property  of 
the  Riccati  matrix  in  conjunction  with  the  Kronecker  sum,  the  number  of  equations  to  be 

solved  can  be  reduced  to  —n  + ^ \ 

Once  a current  value  of  Pk  ( =P(tk ))  is  available,  the  new  solution  is  simply 

pk+ 1 = Pk  + PA  (3-17) 

where  uhk”  is  the  current  integration  step  size.  After  obtaining  Pk+X,  let  k = k + 1 and 
repeat  the  process  to  find  a new  Pk+X  until  tk  = 1.  Then,  solving  an  algebraic  Riccati 
Equation  is  equivalent  to  following  the  homotopy  path  equation  (Eq.  (3.14))  (from  t = 0 
t°  t = 1)  with  a “positive”  starting  point,  P0,  which  is  the  inverse  of  the  solution  of  the 
following  Lyapunov  equation 

(A  + ar)V  + V(A  + aI)T  - BR~'Bt  = 0 (3.18) 

Notice  that  the  homotopy  differential  equation  (Eq.  (3.16))  is  in  the  form  of  a system 
of  linear  uncoupled  first  order  ordinary  differential  equations  for  P.  Several  numerical 
methods  (e.g.,  Euler  method,  Runge-Kutta  methods,  predictor-corrector  methods,  etc.)  can 
be  utilized.  Numerical  integration  using  Euler  method  is  not  very  accurate  when  the 
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integration  step,  hk,  is  too  large  (i.e.,  error  at  each  integration  step  is  0 (hf)).  Hence,  some 
of  the  existing  homotopy  algorithms  utilize  a correction  formula  (e.g.,  Newton’s  formula) 
to  reduce  the  integration  error  [36,41],  Instead  of  using  those  predictor-corrector  methods, 
the  proposed  homotopy  algorithm  uses  a variable  step  size  Runge-Kutta  code  (i.e.,  ode45.m 
in  MATLAB)  to  implement  the  numerical  integration.  Numerical  results  show  the  solution 
is  fairly  accurate  (numerical  examples  are  in  Section  3.4).  It  should  be  noted  that  in  solving 
the  reduced  form  of  Eq.  (3.16),  the  Jacobian  matrix  is  QR  factorized  and  back  substituted 
at  each  integration  step  to  improve  numerical  efficiency. 

3.3.1  Positive  Definite  Solution  and  Path  Existence 

A detailed  proof  that  the  “Jacobian  matrices”  are  non-singular  at  each  integration  point 
on  the  homotopy  path  and  the  final  solution  (t  = 1)  is  positive  definite  will  be  presented  in 
this  section. 

Theorem  3.3.1. 1 (Gopal  [44],  pp.  211) 

Given  a time-invariant  system 

x = Ax  + Bu  (3.19) 

where  A E <5b>nxn,  B E %nxm,  the  following  statements  are  equivalent: 

1 . The  system  is  completely  controllable. 

2.  The  rows  of  eAt  B are  linear  independent  over  the  real  field  for  all  t E [0,  oo ). 

3.  The  state  covarience  matrix  is  positive  definite,  i.e., 

00 

V > 0,  V t j eA,BR~xBTeATt  dt,  R > 0 

o 


□ 


4. 


rank  C = n,  C * [B,  AB,  -,  An~xB]. 
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Corollary  3.3.1. 1 (Skelton  [32],  pp.  211) 


Let  R > 0.  If  all  eigenvalues  of  A in  Eq.  (3. 19)  are  negative  definite,  then  rank  C = n 
iff  V > 0 where  V satisfies 

AV  + VAt  + BR~]Bt  = 0 


Corollary  3.3.1.2 


□ 


If  (A,  B)  is  a controllable  pair  and  the  matrix,  A + al,  is  positive  definite,  then  the 
non-trivial  solution,  P,  from  Eq.  (3.13)  is  positive  definite. 

Proof: 


From  Statement  2 in  Theorem  3. 3. 1.1,  if  (A,  B)  is  controllable,  then  the  rows  of  eA  1 B 
are  linear  independent  over  the  real  field  for  all  t E [0,  oo ).  (Notice  that  in  this  proof,  the 
symbol,  t,  stands  for  time  variable,  but  not  for  “homotopy  variable.”)  Based  on  the  matrix 
multiplication  commutativity  between  matrices  At  and  atl, 

e(A+al)tB  = eAteatIB  _ eat^AtB) 

Therefore,  (A  + al,  B ) is  controllable,  since  the  rows  of  eat(eA  ' B ) are  linear  independent 
over  the  real  field  for  all  t €E  [0,  oo).  Based  on  Corollary  3.3. 1.1  and  A + al  being  positive 
definite,  the  controllability  grammian  V is  positive  definite  and  satisfies 

( - A - aI)V  + V(  — A - aI)T  + BR~lBT  = 0 (3.20) 

where  R E %mXm  is  positive  definite.  Substitution  of  P ~ 1 for  V,  and  pre-multiplying  and 
post-multiplying  the  above  equation  by  P yields  Eq.  (3.13).  Hence,  the  solution,  P,  from 
Eq.  (3.13)  is  positive  definite. 

□ 
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It  can  be  shown  that  the  “starting  point”  (i.e.,  solution  of  Eq.  (3.13))  to  the  proposed 
integrative  scheme  is  positive  definite  based  on  Theorem  3.3. 1.1,  Corollaries  3. 3. 1.1  and 
3.3. 1.2. 

Based  on  Corollary  3.3. 1.2,  the  “shifting  parameter,”  a,  is  chosen  as  either  zero  or  a 
positive  random  number  plus  the  absolute  value  of  smallest  non-positive  eigenvalues  of  the 
matrix  A (if  A has  one).  The  reason  to  add  a random  number  to  a is  to  make  the  matrix, 
A + al,  strictly  positive  definite,  thus  the  “starting  point”  to  the  homotopy  differential 
equation  (Eq.  (3.13))  is  positive  definite. 

Notice  that,  from  Corollary  3.3. 1.1,  if  the  solution,  V,  of  the  Lyapunov  equation  (Eq. 
(3.18)  or  (3.20))  has  a non-positive  eigenvalue,  then  the  systems,  ( A,B ) and  (A  -I-  al,B), 
are  not  controllable.  For  the  non-controllable  system,  the  homotopy  algorithm  will  stop  at 
the  “starting  point”  and  give  no  solutions  for  the  algebraic  Riccati  equation,  since  the 
inversion  of  V is  not  positive  definite. 

Up  to  this  point,  the  starting  point,  P0,  of  the  proposed  homotopy  algorithm  is  proven 
to  be  positive  definite.  It  can  also  be  shown  the  starting  Jacobian  matrix  is  non-singular.  The 
following  theorems  are  need  for  the  proof  of  the  Jacobian  matrix  being  non-singular. 

Theorem  3.3.1.2  (Graham  [43],  pp.  30) 

If  {a;}  and  {j3j}  are  all  the  eigenvalues  associated  with  matrices  A and  B,  then 
[a(  + /?;}  are  all  the  eigenvalues  of  A © B. 

□ 


Theorem  3.3.1.3  (Wiberg  [45],  pp.  214) 

If  P is  the  positive  definite  solution  of  the  algebraic  Riccati  equation,  the  closed-loop 
system  is  asymptotically  stable  (i.e.,  the  real  parts  of  all  eigenvalues  of  the  closed-loop 
system  matrix  are  negative). 


□ 
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Based  on  Theorems  3.3. 1.2  and  3.3. 1.3,  the  starting  Jacobian  matrix  in  Eq.  (3.16)  is 
non-singular,  since  the  starting  As  from  Eq.  (3.15),  is  negative  definite  (i.e., 

As  = A + al  - BR~]BtP0  < 0).  Thus  the  integration  cannot  bifurcate  at  the  starting 
point,  and  PQ  and  the  next  solution  P]  exist 

p\  = po  + fyo  (3.21) 

where  h0  is  the  integration  step  at  point  t0  and  P0  is  the  derivative  of  P with  respect  to 
homotopy  parameter,  t,  at  point  tQ  and  satisfies  the  following  equation 

P(As0  + Aj0P0  = - Q + 2aP 0 (3.22) 

where 

AJ()  = A + al  - BR~lBTP0. 

In  order  to  show  under  certain  conditions  that  P,  is  always  positive,  we  introduce  the 
following  theorems. 

Theorem  3.3.1.4  (Gantmacher  [46],  pp.  224) 

All  the  eigenvalues  of  a real  constant  matrix,  A,  have  negative  real  parts,  iff  the  matrix 
equation 

AtP  + PA  = - E 

(with  any  positive  definite  matrix,  E ) has  a positive  definite  solution,  P. 

□ 

Theorem  3.3.1.5  (Molinari  [47]) 

Let  ( A,B ) be  a controllable  pair.  If  A is  non-singular,  then  there  exists  a positive 
semi-definite  solution  P for  the  following  Lyapunov-type  algebraic  equation 

PA  + AtP  - PBR~lBTP  = 0 


such  that 
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Re(.sI)  < Oandsf  = A J with  i = 1,  2,  n 
where  A,-  are  the  open-loop  poles  and  sx  are  the  closed-loop  poles  with  the  state  feedback  gain 
matrix  K = R~XBTP. 

□ 

First,  we  claim  that  if  a = 0 in  Eq.  (3.22),  then  P0  is  positive  (recall  that  a = 0 when 
all  eigenvalues  of  the  system  matrix,  A,  have  positive  real  parts).  Based  on  Eq.  (3.22)  and 
Theorem  3.3. 1.4,  the  proof  is  obvious.  Thus  P x is  positive,  since  Px  = P0  + P0/i0  an<^ 
hQ  > 0.  Now,  if  a > 0 (i.e.,  the  system  matrix,  A,  has  at  least  one  eigenvalue  which  is 
non-positive),  let 

a = c 7n  + d 

where  an  is  the  absolute  value  of  smallest  non-positive  eigenvalue  of  A and  d is  a positive 

1 

random  number.  The  following  theorem  will  show  that  P,  is  positive  definite  if  h0  < ^ for 
a > 0 case. 

Theorem  3.3.1.6 

Let  P j be  the  solution  of  the  homotopy  equation  (Eq.  (3.14))  at  t = tx  (=  h0 ) 

P,(A  + (1  - h0)al)  + (A  + (1  - hQ)aI)TPx 

- PxBR~xBtPx  + h0Q  = 0 (3.23) 

or 

PxASi  + AjPx  = - PxBR~xBtPx  - h0Q  (3.24) 

where 

A,,  = A + (1  - h0)al  - BR~xBtPx  (3.25) 

and  a = on  + <3  > 0,  where  on  > 0 is  the  absolute  value  of  smallest  non-positive 
eigenvalue  of  A (if  A has  any).  Then  Px  is  positive  definite  if 


ho<  a 
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Proof: 

Define  Aa  as  the  initial  starting  open-loop  system  matrix 

Aa  = A + al 

and  let  S = BR~]BT.  The  solution  from  the  first  integration  step  is  symboled  as 

P\  — 

Substitution  of  Aa,  S and  P j into  Eq.  (3.23)  yields  as 

(^o  + APo)(.Aa  ~ hQaD  + (A a — h0anT(P o + APq) 

- (P0  + AP0)S{P0  + AP0 ) + h0Q  = 0 (3.26) 

Subtract  Eq.  (3.20)  from  Eq.  (3.26) 

— 2h(fiLPQ  — 2h(fLAPQ  -I-  APQAa  + AaAP0 

- PqSAPq  - A PqSP0  - AP0SAP0  + h0Q  = 0 (3.27) 

and  multiply  Eq.  (3.22)  by  hQ 

AP0(Aa  - SP0)  + {Aa  - SP q)tAP o = h0{-Q  + 2 aP0)  (3.28) 

Subtract  Eq.  (3.28)  from  (3.27) 

- 2h0aAP0  = A P0SAP0  (3.29) 

From  Eq.  (3.28),  we  can  show,  based  on  Theorem  3. 3. 1.4  and  Aa  — SP0  < 0,  that 

(1)  AP0  > 0,  if  - Q + 2aP0  < 0 

(2)  AP0  < 0,  if  - Q + 2 aP0  > 0 

In  Eq.  (3.22),  if  we  can  choose  a proper  “a”  such  that  Q — 2 aP0  > 0 or  Q — 2 aP0  < 0, 
then  the  inversion  of  AP0  exists.  Hence,  Eq.  (3.29)  can  be  written  as 

- 2 h0al  = APqS. 

Substitution  of  “P0  + APq  for  Pj,  2/z0aP’  for  SAP0,  and  “BR~lBT”  for  S in  Eq. 
(3.25)  yields 

A5j  = AS{)  + h0al.  (3.30) 

According  to  Molinari’s  mirror  image  property  (Theorem  3.3.1 .5),  the  largest  eigenvalue  of 
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ASq  is  — d,  since  the  smallest  eigenvalue  of  Aa  is  <5.  Hence  if 

K < I-  (3.31) 

then  ASi  is  negative  definite.  From  Eq.  (3.24)  and  Theorem  3.1.1,  one  can  show  that  P]  is 
positive  definite. 

□ 

We  have  shown  that  if  either  a = 0 or  Eq.  (3.31)  holds  (but  not  both),  then  P,  is 
positive  definite.  For  a positive  definite  P,,  one  can  prove  that  Px  exists  from  Eq.  (3.16) 
and  Theorem  3. 3. 1.3.  Any  successive  integration  solution  on  the  homotopy  path  described 
by  Eq.  (3.14)  is  guaranteed  to  exist  and  to  be  positive  definite,  and  its  associated  Jacobian 
matrix  is  also  guaranteed  to  be  non-singular,  based  on  the  following  proposed  theorem. 

Theorem  3.3. 1.7 

The  necessary  condition  for  new  solution,  Pk+  j,  in  Eq.  (3.14)  to  be  positive  definite 
and  the  new  Jacobian  matrix,  Ajk+]  © Ajk+t,  to  be  non-singular,  where 

^k+\  = Pk  + P/ftk 

Ask+i  = A + (1  - tk+l)al  - BR~lBTPk+l 

is  that  current  integration  step  size,  hk,  is  less  than  current  integration  point,  tk,  where  k > 1 . 
Proof: 

We  need  to  consider  two  possibilities:  a = Oanda  > 0.  Fora  = 0,  Pk+]  is  positive 

definite,  since  Pk  (from  Eq.  (3.15))  and  Pk  are  positive  definite,  and  hk  is  positive.  The 

Jacobian  matrix,  Ajk+i  © Ajk+i,  is  non-singular  (more  precisely  negative  definite),  based  on 
Theorem  3. 3. 1.4. 


For  a > 0,  we  first  write  Eq.  (3.14)  at  tk 
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Pk(A  + (1  - tk)al)  + (A  + (1  - tk)aI)TPk 

-PkBR~'BTPk  + tkQ  = 0 


(3.32) 


and  Eq.  (3.15)  at  tk 


P^Sk  + A*  Pk  = -Q  + 2aPk 


(3.33) 


where 


ASk  = A + (1  - tk)al  - BR~xBTPk 


Then,  multiply  Eq.  (3.22)  by  hk  and  add  the  result  to  Eq.  (3.32) 

Pk+\A*k  + AJA+i  = - PkBR~xBTPk  - tkQ  + hk(  - Q + 2 aPk)  (3.34) 

Note  that  from  Eq.  (3.32),  for  tk  * 0 (since  k > 1), 

~ Q + 2 aPk  = y[Pk{A  + al  - BR~lBTPk ) + (A  + al  - BR~{BTPk)TPk  + 
PkBR~xBTPk\ 

Substitute  the  right  hand  side  (R.H.S.)  of  the  above  equation  for  - Q + 2 aPk  into  Eq. 


Notice  that  the  necessary  condition  for  the  R.H.S.  of  the  above  equation  to  be  negative 
definite  is  hk  < tk.  In  other  words,  if  the  current  integration  step,  hk,  is  less  than  current 
integration  point,  tk,  then  the  R.H.S.  of  the  above  equation  is  negative  definite.  For  a proper 
choice  of  hk,  (i.e.,  hk  < tk ),  Pk+\  is  positive  definite,  based  on  Theorem  3. 3. 1.4,  since  the 
R.H.S.  of  the  above  equation  is  negative  definite  and  ASk  is  also  negative  definite.  Moreover, 
Pk+\  is  the  “positive”  solution  of  the  homotopy  equation  (Eq.  (3.14))  at  tk+l,  i.e., 


(3.34) 


+ j-[Pk(A  + al  - BR~xBTPk)  + 


(A  + al  - BR-]BTPk)TPk) 


Sk+\  k+ 1 


Pk+\BR  lBTpk+ 1 h+\Q 


(3.35) 
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where 

A*,.,  = A + (1  - <*+1)a  -BR-'BtPm 

It  can  be  further  concluded  that  AJjfc+i  is  negative  based  on  Theorem  3. 3. 1.4,  since  Pk+l  is 
positive  definite  and  the  R.H.S.  of  Eq.  (3.35)  is  negative  definite.  The  Jacobian  matrix, 
AJk+l  © Ah+r  is  non-singular  (more  precisely  negative  definite),  based  on  Theorem  3.3. 1 .2 

□ 

3.3.2  A Robust  Homotopv  Algorithm 

The  difference  between  the  proposed  algorithm  and  any  other  generic  homotopy 
algorithm  is  that  the  solution  is  guaranteed  to  be  positive  definite.  The  path  following 
algorithm  is  summarized  as  follow: 

Step  1.  Transform  the  algebraic  Riccati  equation  (Eq.  (3.11))  into  the  homotopy 
equation  (Eq.  (3.14)). 

Step  2.  Differentiate  the  homotopy  equation  (Eq.  (3.14))  to  get  the  homotopy 
differential  equation  (Eq.  (3.15)). 

Step  3.  Vectorize  Eq.  (3.15)  to  get  Eq.  (3. 1 6). 

Step  4.  Determine  a starting  point  from  the  inverse  of  solution  of  Eq.(3. 1 8).  Notice 
that  if  the  solution  of  Eq.(3.18)  has  any  non-positive  eigenvalue  (i.e.,  the 
system  is  not  controllable),  then  go  to  step  10  and  stop. 

Step  5.  t0  = 0. 

Step  6.  QR  factorize  current  reduced  (—■  + ^ x n^n  9+  }),  instead  of  n2  X n 2) 

Jacobian  matrix,  Aj  © Aj,  in  Eq.  (3.16),  and  solve  for  Pk. 

Step  7.  Use  a modified  Runge-Kutta  variable  step  size  algorithm  to  numerically 
integrate  Eq.  (3.16).  (Note:  When  new  solution  is  not  positive  definite, 
change  the  integration  step  size  such  that  (i)  current  integration  step,  hK,  is  less 
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than  the  current  homotopy  variable,  tk,  and  (ii)  h0  < if  the  system  matrix, 
A,  has  any  eigenvalue  which  has  a non-positive  real  part). 

Step  8.  tk+l  = tk  + hk  and  P k+ 1 = Pk  + P^ik,  where  hK  is  the  variable  sized 
integration  step  generated  from  “ode45.m.” 

Step  9.  Check  if  tk+ 1 < 1 . 

If  true,  let  k = k + 1 , go  to  step  6,  otherwise,  go  to  step  10. 

Step  10.  Stop. 


3.4  Numerical  Examples 


To  demonstrate  the  robustness  of  our  homotopy  algorithm,  several  examples  are  given. 
Example  3.4.1:  A non-controllable  system  (Wang  and  Pan  [9]) 
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0 . 000  62 . 369  ' 

- 20 . 034  - 57  . 560 

24.011  0.000 

0 . 000  0 . 000 

0 . 000  0 . 000 


, R = Diag([  1 1]),  Q = Diag([\  1112]) 


The  open-loop  system  matrix  A has  eigenvalues  of  - 1 . 1497  ± 13  . 9866/,  - 1 . 0497, 
~ 18  • 6359,  and  0.  Since  matrix  A has  non-positive  eigenvalues,  d and  a are  chosen  to 
be  0 . 2191  (a  positive  random  number)  and  18  . 8548,  respectively.  The  systems,  (A,  B) 
and  (A  + aI,B),  are  not  controllable  and  the  rank  of  their  associated  controllability  matrix 
C(A,  B)  is  4.  Conventional  homotopy  algorithms  [9,10]  cannot  automatically  check  if  the 
system  is  controllable.  Thus,  they  will  generate  a “solution”  to  the  algebraic  Riccati 
equation,  even  though  the  system  is  not  controllable.  The  proposed  homotopy  algorithm 
detects  this  non-controllable  situation  at  the  first  step,  since  the  Lyapunov  equation  (Eq. 
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(3.18)  or  (3.20))  has  a non-positive  eigenvalue  (i.e.,  the  eigenvalues  for  this  Lyapunov 
solution  are  0,  1 . 0368  X 1(T2,  1 . 0883  X 101,  4 . 1621  x 102,  1 . 2651  x 106). 
Example  3.4.2:  A controllable  open-loop  system  with  positive  eigenvalues 
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R = Diag([  1 1]),  Q = Diag([l  111]) 

Matrix  A has  eigenvalues  of  [4 . 3429,  2 . 4707,  2 . 0000,  0 . 1864],  Since  matrix  A has 
no  non-positive  eigenvalues,  both  d and  a are  chosen  to  be  0.  The  pair  (A,  B)  is  controllable. 
The  positive  definite  solution,  P,  is 

"10.5367  3.7065  18.3878  37.1000" 

3 . 7065  8 . 3179  20 . 6206  17  . 8166 

18  . 3878  20 . 6206  1 12 . 6704  135  . 1987 

37.1000  17.8166  135.1987  224.7066 

This  “efficient”  homotopy  scheme  takes  0 . 6179  mega  flops  (using  “flops”  command  in 

MATLAB)  to  reach  the  solution  P.  The  2-norm  of  f(P)  is  3 . 5422  x 10 ~7,  where 

f(P)  = PA  + AtP  - PBR  ]BtP  + Q.  It  should  note  that  the  other  homotopy  scheme, 
which  dose  not  use  symmetric  property  of  the  Riccati  solution,  requires  0 . 7354  mega  flops 
to  obtain  the  solution. 


Example  3.4.3:  A controllable  open-loop  system  with  non-positive  eigenvalues 


1 

1 

N> 

O 

"1 

01 
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1-2  1 0 

0 1-2  1 

, B = 

0 1 
0 0 

1 0 1-2 

0 

0 

R = Diag([  1 1]),  Q = Diag([  1 1 1 1]). 

Matrix  A has  eigenvalues  of  [0,  — 2,  — 2,  - 4].  Since  matrix  A has  non-positive 
eigenvalues,  d and  a are  chosen  to  be  0 . 8977  (a  positive  random  number)  and  4 . 8977, 
respectively.  The  systems,  (A,  B)  and  (A  + al,  B ),  are  controllable.  The  positive  definite 
solution,  P,  is 
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‘0 . 4895  0 . 3085  0 . 2785  0 . 3378' 

0 . 3085  0 . 4895  0 . 3378  0 . 2785 
0.2785  0.3378  0.5561  0.3703 

0.3378  0.2785  0.3703  0.5561 

The  “efficient”  homotopy  scheme  takes  1 . 9454  mega  flops  to  reach  the  solution  P.  The 
2-norm  of  f(P)  is  4 . 6824  X 10-6,  where  f(P)  = PA  + ArP  - PBR~XBTP  + Q. 
Notice  that  the  other  homotopy  scheme,  which  does  not  use  the  symmetric  property  of  the 
Riccati  solution,  requires  2 . 2834  mega  flops  to  obtain  the  solution. 

3.5  Summary 

In  this  chapter,  a robust  homotopy  algorithm  to  which  solves  the  algebraic  Riccati 
equation  is  presented.  A brief  introductory  for  homotopy  theory  is  first  given.  Next,  a proof 
is  provided  to  show  that  with  proper  selections  of  integration  step  sizes,  the  proposed 
homotopy  algorithm  is  guaranteed  to  generate  the  positive  definite  solution  of  the  algebraic 
Riccati  equation.  The  proposed  homotopy  algorithm  utilizes  the  symmetric  property  of 
Riccati  matrix  in  conjunction  with  the  Kronecker  sum  to  make  itself  more  efficient.  The 
proposed  homotopy  is  robust  in  the  sense  that  (1)  by  proper  selection  of  the  integration  step 
size,  bifurcation  will  never  occur  and  (2)  the  solution  is  guaranteed  to  be  positive  definite 
provided  the  system  is  controllable.  To  this  end,  we  extend  the  application  of  this  research 
to  the  solution  of  the  modified’  or  “coupled”  algebraic  Riccati  equations  in  Chapter  4. 


CHAPTER  4 

AN  EFFICIENT  HOMOTOPY  ALGORITHM  FOR  THE  COUPLED  ALGEBRAIC 

RICCATI  EQUATIONS 

4.1  Introduction 

As  mentioned  in  Chapter  2,  it  has  been  found  that  many  multiobjective  dynamical 
systems  have  conflict  of  interest  between  controller  subsystems  [1,6].  A noncooperative 
differential  game  was  proposed  by  Issaacs  [3]  to  handle  situations  where  conflict  of  interest 
exists.  The  first  research  work  which  constructed  game  theoretic  feedback  controllers  for 
a linear  quadratic  regulator  problem  is  attributed  to  Starr  and  Ho  [4],  In  comparison  with 
all  the  other  game  theoretic  strategies  (i.e.,  Pareto,  Mini-max  and  Stackelberg),  a Nash 
strategy  is  the  only  strategy  in  which  the  state  feedback  controllers  can  exchange  dynamic 
information  explicitly  for  multiobjective  linear  quadratic  regulator  problems.  Thus,  a Nash 
strategy  is  uniquely  suited  for  multiobjective  linear  quadratic  problems  involving  conflict 
among  controllers.  Furthermore,  for  the  infinite  time  horizon  problems,  it  has  been  shown 
that  the  game  theoretic  state  feedback  gain  matrices  are  constant  in  time  and  are  determined 
from  solutions  of  coupled  algebraic  Riccati  equations  [1,5].  As  a clarification,  throughout 
this  dissertation,  the  author  uses  the  phrase  “game  theoretic  state  feedback  controller”  as  the 
abbreviation  of  “a  game  theoretic  state  feedback  controller  using  a Nash  strategy.” 

Numerous  authors  have  contributed  to  solutions  of  the  coupled  algebraic  Riccati 
equations.  For  examples,  an  iterative  scheme  based  on  Newton’s  method  was  developed  by 
Krikelis  and  Rekasius  [5]  to  solve  zero-sum  differential  games.  Another  algorithm,  which 
is  based  on  a conjugate  gradient  technique  to  solve  cooperative  differential  games  was 
formulated  by  Innocenti  and  Schmidt  [7],  Strikant  and  Basar  [48]  proposed  an  iterative 
scheme  to  solve  noncooperative  nonzero-sum  differential  games  with  “weakly”  coupled 
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players.  Later  on,  a modified  Krikelis’s  scheme  which  utilizes  the  properties  of  Kronecker 
products  to  solve  2-player  nonzero-sum  noncooperative  differential  games  with  “strongly” 
coupled  players  was  proposed  by  Fitz-Coy  and  Jang  [ 1 ].  However,  convergency  of  all  these 
iterative  schemes  greatly  depend  upon  a good  initial  guess.  A bad  initial  guess  may  result 
in  these  iterative  schemes  failing  to  converge. 

Homotopy  methods  have  been  proven  and  implemented  as  globally  convergent 
algorithms  to  numerically  determine  the  solutions  of  a system  polynomial  equation  [8], 
Homotopy  methods  can  also  be  utilized  to  solve  the  algebraic  Riccati-type  equations,  since 
these  equations  can  be  converted  into  the  form  of  systems  of  polynomial  equations.  In 
Chapter  3,  we  developed  a robust  homotopy  algorithm  for  the  “classical”  algebraic  Riccati 
equation.  Richter,  Hodel  and  Pruett  [36]  used  a homotopy  method  for  the  numerical  solution 
of  the  modified  algebraic  Riccati  equation.  Collins,  Lawrence  and  Richter  [41]  proposed 
a homotopy  algorithm,  which  is  capable  of  solving  two  Riccati  equations  weakly  coupled 
to  two  Lyapunov  equations.  In  this  chapter,  a homotopy  algorithm  is  proposed  to  solve 
strongly  coupled  algebraic  Riccati  equation  which  results  from  solving  multiobjective  linear 
quadratic  regulators  problems  using  the  Nash  strategy.  The  proposed  algorithm  guarantees 
to  give  a solution  of  coupled  algebraic  Riccati  equations.  The  proposed  homotopy  algorithm 
is  also  very  efficient  since  it  utilizes  the  symmetric  property  of  coupled  algebraic  Riccati 
equations  in  conjunction  with  the  Kronecker  sum. 

This  chapter  is  organized  in  the  following  manner.  In  Section  4.2,  an  efficient  (as 
opposed  to  generic  homotopy  codes)  “linear”  homotopy  algorithm  for  the  solution  of 
coupled  algebraic  Riccati  equations  is  presented.  Numerical  examples  and  a summary  are 
provided  in  Sections  4.3  and  4.4,  respectively. 

4.2  Homotopy  Algorithm  to  Solve  Coupled  Algebraic  Riccati  Equations 

A “strongly”  coupled  algebraic  Riccati  equations  is  defined  as  (as  opposed  to  the 
“weakly”  coupled  algebraic  Riccati  equations  [48]) 
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PXA  + ArP,  + fij  - P^Pf^flJPi  - PXB2R~2XBT2P2 

- P2B2R^BT2PX  + P2B2R22lRl2R22lBlP2  = 0 (4.1) 

^ + ArP2  + Q2  - P2B2R22lBT2P2  - P2BxRxxxBtxPx 

- P]BlR-lB]P2  + PjfljPj-j^jPfj^fP!  = 0 (4.2) 

where  matrices  A,  P}  and  Q-}  G 3^nX”,  B}  G <&nXmJ,  RjX  G Rj2  G 

with  j — 1,2.  Matrices  P;  and  P/;  are  positive  definite,  Qj  is  positive  semi-definite  with 

j = 1,2.  Let 

/,</>„  p2)  = v + a7>,  + e, 

- P,B,R^'B]Pt  - P,B2R^'Bt2P2 

~ P2P2P2"21P[P,  + P2B2R22R\2R22B2P2  = 0 (4.3) 

fl(P]’Pl)  ~ P lA  A-  AtP2  + 2 2 

- P2fl2R2-2'fi!>2  - RjR^r.'sfR. 

- P.BiRfi'BjPj  + P.B.Rf/R^Rfi'BjP,  = 0 (4.4) 

and  choose  “starting  points”  as  nonzero  solutions  of  e](P]lP2)  and  e2(Pj,P2) 

ei(P,,P2)  - PX(A  + a/)  + (A  + a/)rP, 

- P^jPj-'BfPj  - PxB2R22'Bt2Px  = 0,  (4.5) 

e2(Pi,P2)  = P2(A  + a/)  + (A  + a/)rP2 

- P^Pfj'PjPj  - P2B2R2jBT2P2  = 0 (4.6) 

where  a is  either  zero  or  a positive  random  number  plus  the  negation  of  smallest  non-positive 
eigenvalues  of  the  matrix  A.  The  reason  to  choose  such  an  a is  the  same  as  described  in 
Section  3.3.1.  The  coupled  homotopy  equations  become 

hx(Px,P2)  = ex(P j,P2)  + t(fx(Px,P2)  - ex(Px,P2)) 

— Pj(A  + al)  + (A  + a[)TPx 
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- PXBXR~XB\PX  - PXB2R~2XBT2PX 

+ t [Qx  — 2aPx  — P XB2R22B^P 2 ~ P 2^ 2^-22  1 

+ PlB2R~2]BT2Pl  + P2B2R22XRX2R22XBT2P2]  = 0 (4.7) 


h2(PvP2)  = e2{Px,P2)  + t{f2(Px,P2)  - e2(Px,P2)) 

= P2(A  + aP>  + (A  + aI)TP2 
- JVWfflfo  - P2B2R22'BT2P2 
+ HQ2-  2 aP2  - P2B,R-'B]P,  - PIBIR-'B]P1 

+ P2BiR^'B\P2  + P^Rn'R^R-'BTpj  = 0 (4.8) 

Differentiating  both  Eqs.  (4.7)  and  (4.8)  yield  the  following  homotopy  differential  equations 
(or  Davidenko  differential  equation) 


pt\  + \TtP,  +P2\+hlrTip2  = A,, 

pl\  + < P , + P2\  + h2pT2P2  = h2: 

, , d/i  . dh 

where  hip.  = = 1,2)  and  hif  = = 1,2).  These  quantities  are: 


(4.9) 

(4.10) 


\ ~ B \R\\  B\P i + (1  ~ 0(a/  - B2R~2xBt2Px ) - r 52/?-15[P2]t 

*i,2  = ^B 2R22  R \2R22  B^P 2 - fl2/?-|5£P1]7' 

\ - S1/?-'5[P2]7' 

^ - B2R22xBt2P2  + (1  - t)(al  - BxR-lB^P2)  - t BxR~lB^Px]T 

Ri,  ~ 2°pi  + P^^^BlP j + PXB2R22B^P2 

— P2B2R22RX2R221P2P 2 — P \B2P 22  B2P i — Qi 

^2,  = 2 °p2  + PxBxR~xB\P2  + P2BXR-XB\PX 

~ P,BiRnlR2iRulB\Px  ~ P^iRuB\P2  - Q2 


Now,  define  H and  Vec(P)  as 
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Vec(hx) 
Vec(h2 ) 


(4.11) 


(4.12) 


where  Vec(Pl ) and  Vec(P2 ) are 


Notice  that  Eqs.  (4.9)  and  (4.10)  are  in  the  form  of  coupled  Lyapunov  equations,  which 
cannot  be  solved  with  standard  Lyapunov  solvers.  In  order  to  solve  these  coupled  Lyapunov 
equations,  a vectorization  procedure  similar  to  that  described  in  Chapter  3 has  to  be 
performed  [6,49].  Equations  (4.9)  and  (4.10)  must  be  vectorized  and  are  rewritten  in  the 
matrix  equation  form: 


Hence,  solving  the  coupled  algebraic  Riccati  equations  (Eqs.  (4.1)  and  (4.2))  is  equivalent  to 
finding  the  solution  of  Eq.  (4. 13)  with  two  sets  of  starting  points  from  non-zero  solutions  of 
the  following  uncoupled  Lyapunov  equations 

(A  + aI)P~l  + P~X(A  + aI)T  - BXR~XB\  - B2R~2XB2  = 0 (4.14) 

(A  + aI)P~x  + P~\A  + aI)T  - BXR~XBTX  - B2R~XB2  = 0 (4.15) 

This  derivation  and  procedure  of  the  proposed  homotopy  algorithm  for  coupled 

algebraic  Riccati  equations  is  similar  to  the  one  for  the  algebraic  Riccati  equation  that  was 

presented  in  Chapter  3.  Both  algorithms  starts  with  the  solution  of  a Lyapunov-type 
equation,  and  then  integrates  the  Lyapunov-type  differential  equation  over  the  homotopy 
variable  t.  However,  the  homotopy  algorithm  for  the  solution  of  coupled  algebraic  Riccati 
equations  involves  the  integration  of  coupled  differential  equations.  This  coupling  effect 


(4.13) 
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complicates  the  analysis  on  the  positive  definiteness  of  the  solution.  Thus,  unlike  the 
algorithm  presented  in  Chapter  3,  this  proposed  homotopy  algorithm  was  not  analyzed  for 
the  positive  definiteness,  hence  there  is  no  guarantee  that  the  solution  to  the  coupled 
algebraic  Riccati  equations  will  be  positive  definite. 

The  proposed  homotopy  algorithm  is  more  efficient  than  the  generic  homotopy  codes. 
This  improvement  in  efficiency  results  from  using  the  symmetric  property  of  the  coupled 
algebraic  Riccati  equations  in  conjunction  with  the  Kronecker  sum.  Also,  as  was  shown  in 
Chapter  3,  the  “integration  path”  will  “never”  bifurcate  (i.e.,  bifurcation  points  can  be 
bypassed  along  the  homotopy  path),  according  to  the  Sard’s  theorem  (Theorem  3.2.3  and 
Corollary  3.2.3). 

4.3  Numerical  Examples 

Two  examples  are  presented  to  demonstrate  the  computational  efforts  vs.  convergence 
guarantee  for  solving  the  two-person  non-cooperative  non-zero  sum  differential  game 
problems. 

Model: 

Given  a dynamical  system  as  follow 

X = Ax  + 5|Mj  + ^2m2 

Find  a pair  of  feedback  controllers,  and  u2,  1°  minimize  the  following  performance 
indices  simultaneously 

00 

min  i f T T 

Uj  Jj  = ^ (x^QjX  + u\Rj]ul  + uJ2Rj2u2 ) dt,  j = 1,2 
0 

where 

uj  = - j = 1,2 

Example  4.1.  The  example  is  selected  from  an  isolation  problem  [6]. 
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Both  the  Newton-iterative  scheme  [ 1 ] and  the  proposed  homotopy  scheme  are  used  to  solve 
this  example.  Both  schemes  lead  to  the  same  positive  definite  solution  pair 
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The  Newton-iterative  scheme  has  a stopping  criterion  that  2-norm  of  both  fx{Px,P2 ) and 
fi^-Pi'P?)  are  less  than  10  7,  where  fx(Px,P2)  and  f2(Px,P2)  are  defined  in  Eqs.  (4.3)  and 
(4.4),  respectively.  By  using  the  proposed  homotopy  algorithm,  the  2-norm  of  fx  (Px , P2)  is 

1 .6307  x 10 _ 7 and  the  2-norm  of  f2(Px,P2)  is  1 .5231  x 10-6.  The  scheme  based  on 
the  Newton-iterative  method  takes  227,838  flops  (using  “flops”  command  in  MATLAB)  to 
reach  the  solution,  and  the  “efficient”  homotopy  scheme  takes  46 . 8228  mega  flops.  (The 
other  homotopy  scheme,  which  does  not  use  the  symmetric  property  of  the  Riccati  solutions, 
takes  234 .6111  mega  flops.)  Notice  that  mega  flops  is  the  unit  used  when  evaluating  the 
floating  points  operation  for  a homotopy  scheme  (e.g.,  see  Reference  50).  This  example 
shows  that  if  the  iterative  scheme  converges,  the  iterative  scheme  is  typical  more  efficient 
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than  the  homotopy  scheme.  To  demonstrate  a convergence  failure  of  the  Newton-iterative 
scheme,  we  examine  the  next  example. 

Example  4.2.  This  example  shows  that  only  the  homotopy  scheme  yields  solutions. 
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Q\  = Diag([]  0 1 0]),  Q2  = Diag{[ 0 1 0 1]), 


*11  = Diagdl  1]),  Rn  = Diag([ 0 0]),  R2]  = Diag{[ 0 0]),  R22  = Diag([  1 1]). 

Both  Newton-iterative  scheme  [ 1 ] and  the  proposed  homotopy  scheme  are  used  to  solve  this 
example.  However,  only  the  homotopy  scheme  yields  the  solution  pair, 


*1 


4 . 5497  0 . 9949 

0 . 9949  0 . 8287 

0 . 7060  0 . 7205 

1 . 2931  - 0 . 7285 


0.7060  1.2931 

0 . 7205  - 0 . 7285 
1 . 8938  - 1 . 9439 
1 . 9439  2 . 2724 


*2 


0 . 4732 
1 . 2043 
- 0 . 8321 
0 . 6433 


1 . 2043 
3 . 8848 
2 . 7721 
0 . 9137 


0 . 8321 
2 . 7721 
2 . 2196 
0 . 1944 


0 . 6433 
0 . 9137 
- 0 . 1944 
2 . 2560 


the  Newton-iterative  scheme  never  converges  to  the  solution.  The  homotopy  scheme  takes 
36  . 1058  mega  flops  to  get  the  solution.  (The  other  homotopy  scheme,  which  does  not  use 
the  symmetric  property  of  the  Riccati  solutions,  takes  195 . 2124  mega  flops.)  The  2-norm 
of  /,  is  1 . 4817  x 10-9  and  the  2-norm  of  f2  is  2 . 3112  x 10-9. 


4.4  Summary 


This  chapter  introduces  a new  homotopy  scheme  to  solve  strongly  coupled  algebraic 
Riccati  equations.  The  proposed  algorithm  is  more  efficient  than  a generic  homotopy 
scheme,  since  it  utilize  the  symmetric  property  of  solutions  of  Riccati  matrix  in  conjunction 
with  the  Kronecker  sum.  The  convergence  of  the  homotopy  algorithm  to  a solution  of  the 
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coupled  algebraic  Riccati  equations  is  guaranteed  by  the  Sard’s  Theorem.  Two  numerical 
examples  are  given  and  solutions  attempted  by  both  iterative  (Newton)  and  integrative 
(homotopy)  schemes.  Only  the  homotopy  scheme  was  successfully  in  both  examples.  It 
should  be  noted  that  in  general  integrative  schemes  require  more  floating  point  operation 
than  iterative  schemes.  However,  unlike  the  iterative  scheme  which  may  fail  to  converge 
to  a solution,  the  integrative  homotopy  scheme  is  guaranteed  to  yield  a solution. 

Although  the  developed  algorithm  is  capable  of  solving  strongly  coupled  algebraic 
Riccati  equations,  its  utilization  in  solving  real  control  problems  may  be  limited.  These 
limitations  in  application  results  from  ( 1 ) too  much  computational  effort  is  required  to  design 
state  feedback  controllers,  even  for  the  2-player  case,  and  (2)  it  is  very  difficult  to  choose 
input  and  state  weighting  matrices  in  the  performance  indices  to  satisfy  system  design 
criteria  (e.g.,  settling  time,  overshoot  of  the  state  time  response,  etc.).  In  Chapters  5 and  6, 
an  inverse  procedure  (i.e.,  given  desired  closed-loop  poles  and  input  weighting  matrices, 
determine  the  state  weighting  matrices  and  the  feedback  gain  matrices)  is  proposed  to 
overcome  the  above  limitations. 


CHAPTER  5 

DESIGN  OF  UNCONSTRAINED  AND  CONSTRAINED  GAME  THEORETIC 
CONTROLLERS  WITH  PRESCRIBED  EIGENVALUES 

5.1  Introduction 

Current  researches  on  solving  the  differential  game  control  have  all  concentrated  on 
determining  feedback  gain  matrices  for  the  2-player  differential  game  with  given  state  and 
control  weighting  matrices  (i.e.,  solving  a system  of  2-coupled  algebraic  Riccati  equation). 
For  a general  n-player  differential  game,  a solution  scheme  is  still  not  available  due  to  the 
complexity  and  difficulty  in  solving  n-coupled  algebraic  Riccati  equations  numerically.  In 
addition,  based  on  these  approach,  there  is  no  systematic  method  for  obtaining  desired  pole 
locations  for  the  closed-loop  system. 

Since  the  settling  time  of  a stable  linear  system  is  strongly  dependant  on  the  location 
of  the  poles  of  the  system,  pole  placement  via  state  feedback  controllers  has  been  an  active 
area  of  research.  For  example,  Bass  and  Gura  [5 1 ] used  a determinant  identity  technique  to 
obtain  feedback  gains  such  that  a single  input  linear  system  has  desired  eigenvalues. 
Ackermann  [52]  proposed  a pole  placement  technique  for  a multiple  input  linear  system  by 
utilizing  the  “Ackermann’s  formula”  (i.e.,  k = qTna{A),  where  k is  the  state  feedback  gain, 
q„  is  the  last  row  of  the  controllability  matrix,  a(  is  the  coefficients  of  closed-loop 
characteristic  polynomial  equation,  and  a(A)  = An  + a]An~]  + ...  + anI).  By  utilizing 

a pseudo  inversion  technique,  Andry,  Shapiro,  and  Chung  [53]  were  able  to  assign  the 
eigenstructure  of  a multiple  input  linear  system  . 

By  selection  of  various  state  feedback  controllers,  it  is  possible  that  the  resulting 
closed-loop  systems  will  have  identical  sets  of  the  closed-loop  poles;  however,  each 
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controller  may  result  in  an  entirely  different  transient  response  (e.g.,  see  numerical  examples 
in  Chapter  6).  The  methodology  of  “optimal  pole  placement”  which  ties  pole  placement  with 
optimal  control  was  developed  to  choose  the  “optimal  controller”  among  those  designations 
for  the  linear  regulator  problem.  Optimal  pole  placement  was  first  modeled  by  Kalman  [54] 
as  follow  “given  a control  law,  find  all  the  performance  indices  for  which  this  control  law 
is  optimal.”  Numerous  researches,  based  on  Kalman’s  original  definition,  have  been 
developed  to  design  linear  quadratic  regulators  with  a prescribed  set  of  eigenvalues.  For 
example,  given  the  control  weighting  matrix  and  the  desired  real  part  of  the  closed-loop  pole, 
Solheim  [55],  Graupe  [56],  and  Amin  [57]  presented  different  methods  to  find  state 
weighting  and  feedback  matrices  to  form  a closed-loop  system.  Later,  Saif  [58],  Arar  and 
Sawan  [59]  proposed  schemes  which  move  a pair  of  complex  open-loop  poles  to  another  pair 
of  pole  location  (a  pair  of  complex  or  two  real  poles)  for  a closed-loop  system. 

Current  optimal  pole  placement  approaches  are  based  on  the  assumption  that  the  state 
feedback  controllers  optimize  a linear  system  with  a single  objective.  In  this  chapter,  we 
propose  a new  optimal  pole  placement  methodology  based  on  differential  game  theory  for 
the  general  multiple  objective  control  problems.  By  using  multiple  objective  functions,  it 
can  expand  the  control  domain  and  thus  allow  the  control  system  designer  the  freedom  of 
implementing  additional  performance  criteria.  Furthermore,  to  the  best  of  this  author’s 
knowledge,  the  issue  of  weighting  matrices  selection  for  a given  set  of  closed-loop  poles  in 
linear  quadratic  differential  games  has  never  been  addressed  in  the  literature.  The  method 
proposed  in  this  chapter  is  capable  of  shifting  either  a single  pole  or  a pair  of  poles  (two  real 
poles  or  a pair  of  complex  conjugate  poles)  to  the  desired  location(s)  while  satisfying 
additional  design  specifications  for  the  linear  system.  Multiple  poles  shifting  tasks  can  be 
implemented  by  utilizing  the  shifting  algorithm  for  a single  pole  or  a pair  of  poles 
recursively. 

There  are  several  advantages  (over  the  conventional  approach  to  the  classical  optimal 
control  problems  and  the  differential  games)  of  the  proposed  methodology  [60]: 
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(1) .  The  proposed  scheme  can  determine  state  feedback  controllers  in  a 2-player 

differential  game  without  actually  solving  2-coupled  algebraic  Riccati  equations 
numerically. 

(2) .  The  proposed  scheme  can  easily  be  extended  to  solve  the  general  n-player 

differential  game. 

(3) .  The  proposed  scheme  can  incorporate  an  additional  design  criterion  superimposed 

as  an  optimization  criterion,  e.g.,  the  minimum  of  the  square  of  the  Frobenius  norm 
of  the  state  feedback  gain  matrix  can  be  implemented  by  solving  a nonlinear 
programing  problem  (for  detail  see  Chapter  6). 

(4) .  The  solution  of  the  game  theoretic  controller  is  proven  to  be  at  least  equal  to  or 

better  than  the  solution  of  the  LQR  controller,  since  the  latter  one  belongs  to  the 
solution  set  of  the  former  one  (detail  for  a single  pole  shifting  case  see  Theorem 
5.1  in  Section  5.3.2). 

(5) .  The  proposed  scheme  can  also  incorporate  other  physical  design  constraints  as 

additional  inequality  constraints  for  this  optimization  problem,  e.g.,  the  limitations 
which  may  arise  from  vibration  suppression,  based  on  the  consideration  of  both 
comfortability  and  performance,  on  a vehicle  design  (for  detail  see  Chapter  6). 

This  chapter  is  organized  in  the  following  manner.  The  principles  of  optimum  pole 
placement  for  single  objective  control  problems  is  presented  in  section  2.  Section  3 extends 
the  principles  to  multiobjective  control  problems.  Section  4 summarizes  the  proposed  pole 
shifting  scheme.  The  issue  of  additional  criteria  that  must  be  superimposed  in  the  case  of 
pole  placement  for  multiobjective  control  problems  will  be  discussed  in  Chapter  6. 

5.2  Review  of  an  Optimal  Pole  Placement  for  Single  Objective  Control  Problem 

This  short  review  is  based  on  the  recent  research  by  Arar  [59],  A detail  presentation 
of  this  work  along  with  a correction  of  the  optimization  criterion  for  the  case  of  shifting  a 
pair  of  poles  are  also  presented  in  the  Appendix. 
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Given  a linear  controllable  and  observable  dynamical  system 

x(t)  = Ax(t ) + Bu(t)  (5.1) 

where  u(t)  E represents  control  input  to  the  plant,  and  x(t)  E represents  the 

perfectly  measured  plant  state.  Based  on  the  performance  index 

00 

J = j ( xtQx  + utRu)  dt  (5.2) 

o 

the  state  feedback  controller  is 

u(t)  = - R~lBTPx(t ) (5.3) 

where  P is  the  positive  definite  solution  of  the  following  algebraic  Riccati  equation 

Q + PA  + AtP  - PBR~xBtP  = 0 (5.4) 

Consider  the  following  transformation 

= TTx  (5.5) 

where  x E 9k”  represents  the  plant  states  of  the  “true”  model  (i.e.,  the  full  plant  states), 
xr  E 9*/  represents  the  “reduced”  plant  states,  and  T E %nXr  is  the  set  of  left  eigenvectors 
of  the  open-loop  system  matrix  A associated  with  the  open-loop  poles  which  are  to  be 
shifted.  Applying  this  transformation  to  Eq.  (5.1)  yields  a system  of  equations  which  is 
referred  as  the  “reduced”  model  [61] 

ir(0  = ArxM  + BfUfid)  (5.6) 

where  Ar  E %rXn  and  Br  E <&rXm  are 

TtA  = ArTT  (5.7) 

Br  = TtB  (5.8) 

The  reduced-order  state  feedback  control  law,  ur,  of  the  reduced  system  is 

ur(t)  = - R~]BjPrx,(t)  (5.9) 

Pr  is  the  positive  definite  solution  of  the  reduced  algebraic  Riccati  equation 

Qr  + PAr  + Ajpr-  P,BrR~XBTrPr  = 0 


where 


(5.10) 
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P = TPrTT  (5.11) 

The  reduced-order  state  feedback  gain  matrix  is 

Kr  = R~xBrPr  (5.12) 

which  results  in  the  reduced-order  closed-loop  system 

xM  = (Ar  - BrKr)Xr(t)  (5.13) 

Observe  that  the  characteristic  polynomial  equation  for  the  reduced-order  closed-loop 
system  Ar  — BrKr  has  the  poles  at  their  desired  closed-loop  locations.  For  a single  pole 
shifting  case,  let  the  characteristic  polynomial  equation  for  the  reduced-order  closed-loop 
system  be 

s - a0(a,Br,R,Pr)  = 0 (5.14) 

where  the  quantity  a is  the  open-loop  pole  to  be  shifted.  Assume  that  this  open-loop  pole 
moves  from  its  current  location  to  a new  location,  say  ju,  then  the  characteristic  polynomial 
equation  for  the  reduced-order  closed-loop  system  can  also  be  expressed  as 

s ~ ju  = 0 (5.15) 

By  comparing  coefficients  between  Eqs.  (5.14)  and  (5.15),  Pr  is  obtained  by  solving 

P = a0(a,Br,R,Pr)  (5.16) 

Similarly  for  a pair  of  poles  shifting  case,  let  the  characteristic  polynomial  equation  for 
the  reduced-order  closed-loop  system  to  be 

s2  - ax(ax,a2,BrR,Pr)s  + a0(ava2,BrR,Pr ) = 0 (5.17) 

where  the  quantities  a ] and  a2  are  the  open-loop  poles  to  be  shifted.  Suppose  that  these 
open-loop  poles  move  from  their  current  location  to  new  locations,  say  /u , and  ju2,  then  the 

characteristic  polynomial  equation  for  the  reduced-order  closed-loop  system  can  also  be 
written  as 

s2  - (Mi  + p2)s  + P\Pi  = 0 (5.18) 

By  comparing  coefficients  between  Eqs.  (5.17)  and  (5.18),  Pr  is  obtained  by  solving 

P\  + P2  = ax(ax,a2,Br,R,Pr) 

P\Pi  = - a0(al,a1,Br,R,Pr ) 
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5.3  Pole  Placement  for  Multiobjective  Problem 


5.3.1  Preliminary  Transformation 


For  conciseness,  the  following  derivation  is  based  on  the  2-player  differential  game. 
The  n-player  differential  game  problem  is  easily  derived  in  a similar  manner.  (An  example 
is  given  in  Chapter  6.)  Consider  the  following  transformation 

xr=TTx  (5.19) 

where  x,  xrand  T are  defined  in  Section  5.2.  Applying  this  transformation  to  Eq.  (2. 1 ) yields 
a system  of  equations  which  is  also  referred  as  the  “reduced”  model 

xXt)  = A^Xt)  + B{uXr{t)  + B2u2r(t ) (5.20) 

where  Ar  E &rXr satisfies  TrA  = ArTT,B , = TtB , G %rXm\ ,B,  = TtB^  E <3brXm2 

1 r l Zf  Z 

The  performance  indices  associated  with  the  reduced  system  are 

00 

Jir(ulr’u2r)  = J {xTrQirxr+uTxRiXuXr  + uT2Ri2u2)  dt,  i = 1,2  (5.21) 

o 

where  Qir  E 9 lrXr(i  = 1, 2)  is  related  to  the  Qt  ( i = 1, 2)  in  Eq.  (2.2)  by 

Qi  = TQirTT,  i = 1,2  (5.22) 

and  the  control  weighting  matrices  Rn  and  Rl2(i  =1,2)  remains  the  same  as  in  Eq.  (2.2). 
Assuming  the  state  feedback  for  the  reduced  system  results  in  the  following  control  actions 

uir(t)  = ~ R^BjPirxAt),  i = 1,2  (5.23) 

where  Pir,  i = 1,2,  are  the  positive  definite  solutions  of  the  coupled  algebraic  Riccati 
equations  associated  with  the  reduced  model 


e,r  + P,Ar  + Ajp,r  - B,  tfrX/l,  - PKB2,RnBiP2, 

- P2B2RvBlP,r  + P2B2Ri2% 2R22lBlP2r  = 0 (5-24) 
e2r  + P2A,  + Ajplr  - P2B2Rn'BlP2r  - P2  B,  Rn'B\Pu 
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where 


- P\rB\rRUB\,Pl r + Pl ,B  I *fl  ‘^fl  , = 0 (5.25) 


Pi  = TP,TT,  < = 1,2 

The  closed-loop  system  for  the  reduced  model  is 


(5.26) 


= (^r-  I BiR-'BlPijxM 


(5.27) 


i = 1 


For  future  discussion,  we  define  the  feedback  gain  matrix  for  the  reduced  system  as 


Kr  = 


*n'Blpir 

R22R2rR2r 


(5.28) 


The  resulting  closed-loop  system  equation  for  the  true  states  can  be  written  as 


*0)  = (A  - 2 W XBTlrPlrTT)x(t) 


(5.29) 


i = i 


Similarly,  we  define  the  feedback  gain  matrix  for  the  “true”  system  as 


K = KrTT  = 


r~\rt  p 
Kn  Di/\r 

R22R2rR2r 


(5.30) 


It  has  been  shown  by  Rao  and  Lamba  [61]  that  the  n closed-loop  poles  of  Eq.  (5.29)  consists 
of  the  r closed-loop  poles  of  Eq.  (5.27)  and  the  n — r undisturbed  open-loop  poles  of  Eq. 
(2.1).  We  now  present  the  procedure  for  shifting  a single  pole  and  a pair  of  poles. 

5.3.2  Shifting  a Single  Pole 


Let  a be  a single  pole  of  the  linear  system  which  is  to  be  moved  from  its  current  location 
to  a new  location,  say  ft.  From  Eqs.  (5. 1 9),  (5.20)  and  (5.23),  and  choosing  T to  be  the  left 
eigenvector  associated  with  the  left  eigenvalue,  we  get 

P ~ a ~ R\rR\\BT\rP\r  ~ B2rR22B2rR2r  (5.31) 

Notice  that  Eq.  (5.31)  is  an  underdetermined  system  (i.e.,  there  are  fewer  equations  than 
unknowns),  thus  the  solution  of  the  Eq.  (5.31)  is  not  unique.  The  non-uniqueness  of  choice 


63 


for  PXr  and  P2r  gives  the  control  system  engineer  the  ability  to  implement  design 
specifications,  such  as  minimum  envelop  bound  of  the  system  transient  response,  as  addition 
optimality  criteria.  On  the  contrary,  the  classical  “inverse  optimal”  schemes,  (e.g.,  see 
References  57  and  59),  do  not  provide  such  a design  space.  Several  valid  optimization 
criteria  are  presented  in  the  section  4.  Notice  that  the  inequality  constraints  for  the 
optimization  problem  are  Pir  > 0,  (i  = 1,2). 

Comparison  with  a LOR  controller 

For  a LQR  controller,  a and  /i  are  the  associated  open-loop  and  closed-loop  poles, 
respectively,  for  a game  theoretic  controller.  Using  the  results  from  Arar  and  Sawan  [59] 
(modified  version  with  correction  provided  in  the  Appendix),  the  reduced-order  closed-loop 
system  matrix  associated  with  the  LQR  controller  satisfies  the  equality  constraint  equation 


R"1  0 

'B\'r 

P = a - B\r  B2r)  ' 

1 1 i 

0 ^22 
» 4 

- < 

[Bl\ 

>Pr 

(5.32) 

Expanding  Eq.  (5.32)  yields 

P - ° ~ BxRxxB\Pr  - B2R12BT2Pr  (5.33) 

Therefore,  the  solution  to  the  reduced  algebraic  Riccati  equation  is 

P'  = B>rRnK  + (534) 

For  a process  which  is  stabilizing,  (i.e.,  moving  a system  pole  to  its  left  in  the  s-plane), 
it  can  be  shown  that  the  solution  for  the  LQR  controller  is  a subset  of  the  solution  of  the  game 
theoretic  controller  in  a single  pole  assignment  problem.  The  following  theorem  and  a 
numerical  example  are  presented  to  reinforce  this  premise. 

Theorem  5.1 

For  the  case  that  a single  pole  is  shifted  to  its  left  from  its  current  location  in  the  s-plane 
(i.e.,  the  closed-loop  pole  is  more  negative  than  the  open-loop  pole),  the  solution  of  the  LQR 
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controller  (Eq.  (5.34))  belongs  to  the  solution  set  of  the  game  theoretic  controller  (Eq. 
(5.31)). 


Proof: 

For  the  case  of  interest,  a > /u.  Thus  Pr  is  positive,  since  a — /u  is  positive  and 
+ ^2^22  ^\r  are  positive  for  any  non-zero  B (=  [ Bx  B2  ]).  By  substituting 
Pr  for  Plr  and  Pr  for  P2r  into  Eq.  (5.31)  yields 

n — d — BxRx^B\Pr  — 1*2^22  r 

which  is  identical  to  Eq.  (5.33).  Hence,  the  solution  of  the  LQR  controller  (Eq.  (5.34)) 
belongs  to  the  solution  set  of  the  game  theoretic  controller  (Eq.  (5.31)). 

□ 

In  a 2-player  game,  the  solution  set  of  the  game  theoretic  controller  is  a straight  line  in 
the  first  quadrant  of  P\P 2r  plane,  and  the  solution  of  the  LQR  controller  is  a point  on  that 
straight  line  (see  Example  5.1  below). 


Example  5.1: 


Consider  the  following  lateral  model  of  the  F-4  aircraft  [62],  The  controllable  system 
is  given  as 


- 1 . 7680  0 . 4125  - 14 . 2500  0 

- 0 . 0070  - 0 . 3831  6 . 0380  0 

0.0016  - 0.9975  - 0.  1551  0 

1 . 0000  0 . 0000  0 . 0000  0 


0000' 

1 . 7440 

8 . 9520' 

0000 

x + 

- 2 . 9200 

- 0 . 3075 

0586 

0 . 0243 

- 0 . 0036 

0000 

0 . 0000 

0 . 0000 

where 


P' 

roll  rate  ' 

r 

yaw  rate 

f <5r 

* = < 

sideslip 

► and  u = \ 

<P 

roll  angle 

[ a. 

(rudder 
aileron 


The  quadratic  performance  indices  are 
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Ji  = J (xTQiX  + u\Raux  + u\Ri2u2)  dt  (/  =1,2) 

o 

where  /?u  = 1,  Ri2  = 0,  R2 j = 0,  and  R22  = 1.  Partition  the  control  task  results  in 
5j  = [1  . 7440  - 2 . 9200  0 . 0243  0 . 0000]r 

B2  = [8 . 9520  - 0 . 3075  - 0 . 0036  0 . 0000]7' 

In  order  to  compare  the  game  theoretic  state  feedback  controller  with  a LQR  state  feedback 
controller,  the  following  performance  index  is  chosen  for  the  LQR  controller 

OO 

J = J\  + J2  = J (xtQx  + utRu ) dt 
o 

where  the  state  weighting  matrix  Q and  the  control  weighting  matrix  R are 

Q = Q\  + G2’  R = diag([Ru  R22]) 

The  open-loop  poles  for  the  above  system  matrix  of  the  F-4  aircraft  lateral  model  are 
located  at  [ - 0 . 0150,  - 0 . 2149  ± 2 . 4858y,  - 1 . 8614],  These  poles  correspond  to 
the  spiral  mode,  the  Dutch  roll  mode,  and  the  roll  mode,  respectively.  Assume  that  the 
control  task  is  to  move  the  first  open-loop  pole  of  the  system  from  its  current  location  to  a 
new  location  at  —0.5.  The  solution  set  to  the  game  theoretic  controller  and  the  solution 
to  the  LQR  controller  are  shown  in  Figure  5.1.  In  Figure  5. 1 , it  is  clear  that  the  solution  to 
the  LQR  controller  belongs  to  the  solution  set  to  game  theoretic  controller  for  single  pole 
shifting  case.  Notice  that  if  the  compatible  performance  index  is  chosen  differently,  i.e., 

OO 

J = axJx  + a2J2  = j (xtQx  + utRu)  dt,  a - * 1 (!  = 1,2), 
o 

the  solution  to  the  LQR  controller  is  not  the  same  point  shown  in  Figure  5. 1 . However,  the 
solution  to  the  LQR  controller  still  lies  on  the  line  segment,  which  is  the  solution  set  to  the 
game  theoretic  controller. 
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Fig.  5.1  The  solution  to  LQR  controller  belongs  to  the  solution  set  to  game  theoretic 
controller 

5.3.3  Shifting  a Pair  of  Poles 

Let  a + fij,  £ — fij  be  a pair  of  distinct  left  eigenvalues  of  an  n x n system  matrix  A 
which  are  to  be  shifted  from  their  current  location  to  a new  location  at  y + dj,(f>  - dj.  Also, 
let  the  associated  nonzero  left  eigenvectors  be  u ± jv  (or  u and  v for  two  real  poles).  For 
the  open-loop  poles, 

I = a,  if/?  ^ 0 (a  pair  of  complex  conjugate  poles) 

£ ^ a,  if/3  = 0 (two  real,  distinct  poles) 

and  for  the  closed-loop  poles, 

(p  - y,  if  d * 0 (a  pair  of  complex  conjugate  poles) 

(p  ^ y,  if  <3  = 0 (two  real,  distinct  poles) 

The  following  definition  is  helpful  for  the  case  of  shifting  complex  eigenvalues. 
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Definition  5.1  Van  Loan  [63] 


Let  A be  a n x n matrix.  A scalar,  A,  is  defined  as  a left  eigenvalue  of  the  matrix  A 
with  an  associated  nonzero  left  eigenvector  y if  the  following  condition  holds 

yHA  = XyH 


□ 

Now,  let  a ± fij  be  the  complex  conjugate  pair  of  left  eigenvalues  of  the  system  matrix 
A,  with  associated  nonzero  left  eigenvectors,  u ± jv(u,v  G &").  Based  on  the  Definition 
5.1, then 


T • T 
U - jv‘ 

UT  + jvT 


a + P j 0 

0 a - fij 


(5.35) 


To  obtain  the  “real  form”  of  the  left  eigensolution  equation  [64],  Eq.  (5.35)  is  pre-multiplied 
by  a transformation  matrix,  9' 


n 1 1 


which  yields 


where 


(5.36) 


(5.37) 


A = 


a + j 0 

0 a - pj 


= 


a p 

-P  a 


(5.38) 


Eq.  (5.37)  is  called  the  “real  form”  of  left  eigensolution  equation.  Notice  that  Eq.  (5.37) 
was  used  to  define  the  state  matrix  for  the  reduced  model.  Thus,  the  required  transformation 


matrix  is 
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In  the  case  of  shifting  two  distinct  real  poles,  the  transformation  matrix,  T,  has  the  same 
form  as  shown  above.  However,  w and  v are  the  left  eigenvectors  of  the  system  matrix 
corresponding  to  the  two  distinct  real  eigenvalues. 

Returning  to  the  case  of  shifting  a pair  of  poles,  application  of  the  above  transformation 
matrix  to  Eq.  (2.1)  yields 

xAf)  A r£  Xj{f) 

where  the  definitions  in  Eqs.  (5.19),  (5.20)  and  (5.23)  were  used  to  define  the  closed-loop 
system  matrix  Arc,  as 


= \r  - X B,R-  ojPjr 
1=1 


The  matrix  definitions  in  Eq.  (5.39)  are 


Ar  = 


a p 

-p  e 


(5.39) 


(5.40) 


Plr  = 


P2  = 


*1  Z, 

Lzi  y ij 

x2  Z2 

*2  T2J 


In  the  system  matrix  Ar  of  the  reduced  model,  £ 
complex  conjugate  pair  of  poles,  and  £ * a and  $ 
poles.  Now,  let 


(5.41) 

(5.42) 

= a and  /?  ^ 0 for  the  case  of  the 
= 0 for  the  case  of  the  real  distinct 


KRnBl  = 


a\  b\ 

b\  ci 


B2r^22lP2r  - 


a2  b2 
b2  c2 


Substitution  of  A n PXf,  P2/  BxRulB\r,  and  B2R~JbI  into  Eq.  (5.39)  yields 


(5.43) 

(5.44) 


A 


rc 


a axx j bxzx  - d2X2  - b2z2  ft  - axzx  - bxyx  - a2z2  - b2y2 
- P - cxxx  - dxzx  - c2x2  - d2z2  £ - cxzx  - dxyx  - c2z2  - d2y2 


The  characteristic  polynomial  equation  associated  with  An.  is 
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where 


s 2 + mxs  + m2  = 0 


(5.45) 


mi  “ ai*i  + dxyx  + (cj  + bx)zx  + a2x2  + (1$ 2 + 

(b2  + c2)z2  - £ - a (5.46) 

m2  = [ f\y\  + fl>2  +/322  + *1  ] *1  + t Ux2  +f5Z2  + £2  1 >1  “ f\A  + 

[ ~f3X 2 -/^  ~ (/2  +/4)^2  + £3  1 Zi  + [ /6y2  + £4  ] *2  + 

8 5^2  ~ f 5^-2  + £6^2  + £7  (5.47) 

with  /],  f2,  f3,  /4,  /5,  and  /6  defined  as 

/"l  — ~ blxl’  /"2  = a1^2  — ^2cl 

f 3 ~ alc2  ~ <J2C1’  = a2^1  — ^lc2 

•^5  — b2^\  ~ b\d2,  /g  = a2d2  — b2c2 

and£i>  82’  8 3.  £4,  £5,  £6,  and  g7  defined  as 

£1  = C\P  ~ 82  = ~ b\P  ~ dxa 

83  = ~ b\£  + id\  ~ Q\)P  ~ cxa 
84  = cjfi  - a£,  g5  = - b^  - d2a 

86  = ~ b2$  + (d2  ~ a2>P  ~ c2a 


81  = ft1  + a£ 


The  characteristic  polynomial  equation  associated  with  the  closed-loop  system  which  has  the 
desired  pole  locations  (i.e.,  y + dj,<p  - dj ) is 


where 


2 

s + nxs  + n2  = 0 


(5.48) 


- 2y  (complex  conjugate  case) 

— y — (p  (real  distinct  case) 


n~ 


y2  + ( 5 2 (complex  conjugate  case) 
y<p  (real  distinct  case) 


(5.49) 


(5.50) 
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Since  the  requirement  is  to  move  the  closed-loop  poles  to  the  specified  locations,  then  Eq. 
(5.45)  is  equated  to  Eq.  (5.48)  to  obtain 

m , = «j  (5.51) 

m2  = n2  (5.52) 

Equations  (5.51)  and  (5.52)  are  the  only  equality  constraint  equation  for  this  problem. 
Notice  that  Eqs.  (5.51)  and  (5.52)  represent  an  underdetermined  system  (i.e.,  there  are  two 
equations  in  six  unknowns  (jq,  yq,  Zj,  x2,  y2,  and  z2) ).  Since  the  solution  set  of  Eqs. 
(5.51)  and  (5.52)  is  not  unique,  we  must  impose  additional  optimization  criteria  for  this 
problem.  Again,  notice  that  the  inequality  constraint  equations  for  the  optimization  problem 

are  pir  > °>  0 = 1,2).  Possible  candidates  for  additional  optimization  criteria  in  the  case 

of  pole  placement  for  multiobjective  control  problems  will  be  discussed  in  Chapter  6. 

5.4  Recursive  Algorithm 

The  multiple  poles  shifting  problem  using  a game  theoretic  controller  can  be 
implemented  by  utilizing  the  following  algorithm  recursively 

Step  1 : Select  a set  of  eigen  value(s),  A f (a  single  pole  or  a pair  of  poles)  of  the  current 
system  matrix,  A,,  to  be  shifted. 

Step  2:  Find  the  left  eigen vector(s),  v,-,  of  the  current  system  matrix,  A,  associated 
with  the  selected  eigenvalue(s),  At 

Step  3:  Using  the  left  eigenvetor(s),  v,-,  to  form  the  transformation  matrix,  7). 

Step  4:  Form  the  reduced-set  of  system  equation  (Eq.  (5.3 1 ) for  a single  pole  shifting 
and  Eq.  (5.39)  for  a pair  of  poles). 

Step  5:  Using  one  of  the  superimposed  optimization  model  proposed  in  the  Chapter 
6 to  find  optimal  solutions,  P^and  P2r 

Step  6:  Using  Eqs.  (5.26)  and  (5.30)  to  find  solutions,  Pu  and  P2i,  to  the  full-set  of 
the  coupled  algebraic  Riccati  equations,  and  the  state  feedback  gain  matrix, 

K, 
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Step  7:  Form  the  new  system  matrix  by 

A(  + 1 = At  - BKj 

Step  8:  i = i + 1 

Goto  Step  1,  if  further  pole  shifting  is  necessary, 
otherwise,  stop. 

For  the  multiple  shifting  case  (i.e.,  recursively  using  the  above  algorithm),  the  proposed 
algorithm  does  not  provide  a guidance  to  choose  the  order  for  the  shifting  sequence.  For  a 
real  control  problem,  the  selection  of  shifting  sequence  may  result  in  different  optimum 
solutions.  This  phenomenon  can  be  explained  by  the  game  theory  with  the  Stackelberg 
strategy,  since  selecting  the  order  of  pole  shifting  in  optimal  pole  placement  problem  is 
equivalent  to  selecting  the  order  of  importance  for  all  optimization  indices  in  game.  Notice 
that  for  multiple  shifting  case,  the  open-loop  poles  can  also  be  shifted  simultaneously,  which 
can  avoid  selection  of  the  shifting  sequence;  however,  the  unknowns  of  the  optimization 
model  will  be  increased  quadratically. 

5.5  Summary 

In  this  chapter,  the  principle  of  optimal  pole  placement  in  the  game  theoretic  state 
feedback  controller  is  provided  and  a recursive  algorithm  to  implement  pole  placement  task 
by  utilization  of  game  theoretic  controller  is  provide.  For  the  single  pole  shifting  case,  the 
solution  of  the  LQR  controller  lies  at  a line  segment,  which  is  the  solution  set  of  the  solution 
of  the  game  theoretic  controller.  It  infers  that  for  a pair  of  poles  shifting  case,  the  solution 
set  (i.e.,  a line  segment)  of  the  LQR  controller  lies  in  a 4-dimensional  space,  which  is  the 
solution  set  to  the  game  theoretic  controller.  For  both  single  pole  and  a pair  of  poles  shifting 
cases,  the  solutions  to  the  reduced  system  equations  are  not  unique.  Thus,  a superimposed 
optimization  criterion  is  need.  Superimposed  criterion  can  be  chosen  from  system  design 
specifications  . The  detail  of  those  possible  candidates  will  be  introduced  in  Chapter  6. 


Notice  that  in  the  optimal  pole  placement  with  game  theoretic  controller  problems,  the 
inequality  constraints,  Pir  > 0,  is  a consequence  of  the  “stabilizing”  procedures  (i.e., 
moving  system  open-loop  poles  from  their  current  location  to  the  left  in  s-plane).  For  the 
“de-stabilizing”  task  (i.e.,  moving  the  system  open-loop  poles  from  their  current  location  to 
the  right  in  s-plane),  the  inequality  constraint  can  be  released.  Thus,  for  the  later  case,  there 
is  no  inequality  constraints  imposed  on  Pt  . 


CHAPTER  6 

SUPERIMPOSED  OPTIMIZATION  CRITERIA  ON  DESIGNING  GAME 
THEORETIC  STATE  FEEDBACK  CONTROLLERS 

6.1  Introduction 

In  chapter  5,  a new  pole  placement  methodology  for  game  theoretic  state  feedback 
controllers  is  proposed.  There,  it  was  shown  that  by  using  a game  theoretic  controller  design, 
the  design  space  could  be  expanded  beyond  that  allowed  by  the  LQR  design.  It  was  also 
shown  that  unlike  the  LQR  approach  with  a single  equation  in  one  unknown  (i.e.,  a single 
pole  shifting  case),  an  underdetermined  system  of  equations  resulted  from  the  game  theoretic 
approach.  Thus,  in  order  to  determine  an  unique  solution  for  the  game  theoretic  controller, 
additional  criteria  must  be  superimposed.  These  criteria  could  be  physical  of  artificial 
constraints.  For  example, 

(1) .  Minimize  the  square  of  the  Frobenius  norm  of  the  reduced-order  state  feedback 

gain  matrix. 

(2) .  Minimize  the  square  of  the  Frobenius  norm  of  the  full-order  state  feedback  gain 

matrix. 

(3) .  Minimize  the  envelop  bound  of  reduced-order  state  time  response. 

(4) .  Minimize  the  envelop  bound  of  full-order  state  time  response. 

The  Frobenius  norm  of  the  full  (reduced)  state  feedback  gain  matrix  is  a measure  of  the 
magnitude  of  state  feedback  gain  matrix,  which  is  related  to  the  control  action  on  the  full 
(reduced)  model  of  the  system.  The  envelop  bound  of  the  state  time  response  for  the 
full-order  (reduced-order)  system  model  restricts  the  possible  state  time  response  offset  from 
the  set  point.  The  first  three  criteria  are  mathematically  tractable,  and  are  discussed  in  the 
following  sections.  The  fourth  criterion  is  numerically  implementable,  but  is  somewhat 
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intractable,  due  to  the  complexity  in  evaluating  a matrix  exponential.  Beside  these  four 
proposed  criteria,  there  exist  other  useful  design  specifications  (e.g.,  the  magnitude  of 
weighted  square  displacements  and  velocities,  and  the  magnitude  of  weighted  square  control 
effort  [65],  etc.).  However,  the  optimization  criteria  on  either  “magnitude  of  weighted 
square  displacements  and  velocities”  or  “magnitude  of  weighted  square  control  effort”  used 
in  Reference  65.  cannot  be  chosen  as  a superimposed  optimization  here,  because  each  is 
merely  a component  of  the  performance  indices  (i.e.,  Eq.  (2.36)).  Thus,  if  either  one  of  the 
above  criterion  is  used,  then  it  serves  as  a redundant  criteria  to  the  original  multiobjective 
optimization  problem. 

The  following  describes  the  outline  of  this  chapter.  Section  6.2  discusses  the 
optimization  model  based  on  the  minimum  square  of  the  Frobenius  norms  of  both  the 
reduced-order  and  the  full-order  state  feedback  gain  matrices.  Section  6.3  discusses  the 
minimum  envelop  bound  of  the  reduced-order  state  time  response,  and  Section  6.4  discusses 
the  minimization  of  envelop  bound  of  full-order  state  time  response.  Numerical  examples 
to  demonstrate  the  feasibility  of  the  proposed  schemes  are  presented  throughout  this  chapter. 
The  examples  are  based  on  an  F-4  aircraft’s  lateral  direction  model.  There  is  also  another 
numerical  example  to  demonstrate  that  the  proposed  methodology  can  easily  be  extended 
to  the  ra-player  differential  game.  This  example  is  based  a longitudinal  motion  model  of  an 
AIRC  aircraft  with  three  active  controllers  on  board.  This  chapter  is  concluded  with  a 
summary  in  Section  6.5. 

6.2  Minimize  the  Square  of  Frobenius  Norm  of  Reduced-Order  State  Feedback  Gain 

In  this  section,  the  optimization  model  based  on  the  minimization  of  the  Frobenius 
norm  of  the  reduced-order  state  feedback  gain  matrix,  for  both  single-pole  (in  sub-section 
6.2.1)  and  a pair  of  poles  (in  sub-section  6.2.2)  shifting  cases  are  discussed.  The  first-order 
necessary  condition  (or  Kuhn-Tucker  condition)  and  the  second-order  necessary  condition 
will  be  derived  for  the  both  single-pole  and  a pair  of  poles  shifting  cases. 


75 


We  observe  that  the  Frobenius  norm  of  the  full-order  state  feedback  gain  matrix  ( K)  and 
Frobenius  norm  of  reduced-order  state  feedback  gain  matrix  (Kr)  are  related  as 

II K Ilf  = II  KrTT  ||f<  ||  Kr  yi  T ||F  (6.1) 

Thus,  the  minimum  of  Frobenius  norm  of  the  full-order  state  feedback  gain  matrix  is 
bounded  by  the  product  of  the  Frobenius  norm  of  reduced-order  state  feedback  matrix  ( Kr ) 
and  the  Frobenius  norm  of  the  transformation  matrix  (7).  Thus  a discussion  on  the 
minimization  of  the  Frobenius  norm  of  the  full-order  state  feedback  gain  matrix  will  not  be 
presented. 

6.2.1  Shifting  a Single  Pole 

Given  an  n x n system  equation  (Eq.  (2.1)),  the  single  pole  shifting  problem  with 
minimum  square  of  the  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  can 
be  treated  as  the  following  optimization  problem  (KR1): 

Minimize  (CjF^)2  + ( c2P~,  )2 
subject  to 

M = a-  c3Plr  - c4P2r 

p\ r > 0,  P2r  > 0.  (6.2) 

where 

C1  =ll  Ilf  — 0 
c2  =ll  P22P2r  Ilf-  0 
<3  - s,  SfX, 

c4  = B | 

The  first-order  and  second-order  necessary  condition  for  the  existence  of  a solution  to 
the  above  optimization  model  are  derived  and  analyzed  below. 


First-order  Necessary  Condition 
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Theorem  6.1  First-order  Necessary  Condition  (Kuhn-Tucker  Condition)  [66] 

Let  x*  be  a relative  minimum  of  the  following  optimization  problem  (PI) 

Minimize  f(x ) 
subject  to 

h(x)  = 0,  g(x)  < 0 (6.3) 

Supposing  **  is  a regular  point  for  the  constraints  (i.e.,  Vh(x*)  and  Vg(x*)  are  linear 
independent).  Then  there  is  a Lagrange  multiplier,  Xx,  for  the  equality  constraint,  and  a 
Lagrange  multiplier,  X2,  with  X2  > 0 such  that 

Y/X*  ) + X^Vh(x*)  + X2Vg(x*)  = 0 (6.4) 

Six*)  = 0 (6.5) 

□ 

In  Theorem  6.1,  X2  may  be  non-zero  only  if  its  corresponding  inequality  constraint  is 
active  (i.e.,  the  solution  lies  at  the  boundary).  For  the  optimization  model  (KR1),  the 
inequality  constraints  are  strictly  great  than  zero,  therefore  they  are  inactive.  Based  on  this 
argument  and  Theorem  6.1,  the  first-order  necessary  conditions  for  the  optimization  model 
(KR1)  are 

^■c\p\r  + c3^i  = 0 (6.6) 

lc2p2r  + cAXx  =0  (6.7) 

where  Xx  is  a scalar  Lagrange  multiplier  for  the  equality  constraint  Eq.  (5.31). 

Second-order  Necessary  and  Sufficient  Condition 

Theorem  6.2  Second-order  Necessary  and  Sufficient  Condition  [66] 

Suppose  **  is  a regular  point  for  the  constraints  (Eq.  (6.3)).  The  sufficient  (necessary) 
condition  for  x*  to  be  a strict  relative  minimum  point  of  the  optimization  problem  (PI)  is 


77 


V/(; t*)  + A[V/*(jc*)  + XT2Vg{x*)  = 0 

A£g(**)  = 0 

A2  > 0 

and  the  Hessian  matrix  of  the  Lagrange  equation, 

L(x*)  = f{x*)  + X\h{x*)  + Xfax*) 
is  positive  definite  (semi-definite). 


(6.8) 

(6.9) 


□ 

For  a single  pole  shifting  problem,  based  the  first-order  necessary  conditions  (Eqs.  (6.6) 
and  (6.7)),  the  solutions  of  the  reduced-order  coupled  algebraic  Riccati  equation  (Eqs.  (5.24) 
and  (5.25))  are 


with 


c3^1 

2ci 


C4^1 

2 c2 


(6.10) 


X\  - 2{fi  - a)/(^-  + ^) 

The  Hessian  matrix 

2 c\  0 

0 2c\ 

is  positive  semi-definite,  since  c,  > 0 and  c2  > 0.  Therefore,  the  solutions  (Eq.  (6.10)) 
are  relative  minimums  for  the  optimization  problem  (KR1),  since  the  solutions  satisfy  both 
the  first-order  and  the  second-order  necessary  conditions. 

The  non-linear  optimization  problem  (KR1)  is  solved  using  “constr.m”  in  the 
MATLAB’s  Optimization  Toolbox  [67],  This  MATLAB  function  is  developed  using  a 
Sequential  Quadratic  Programming  (SQP)  method.  In  the  SQP  method,  a Quadratic 
Programming  (QP)  subproblem  is  solved  iteratively.  The  Hessian  of  Lagrangian  is 
estimated  and  updated  at  each  iteration  using  the  BFGS  formula  [68,69], 
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Once  the  solution  set,  Plr  and  P2r,  of  the  reduced-order  coupled  algebraic  Riccati 
equation  is  obtained,  the  reduced-order  state  weighting  matrices,  QXf  and  Q2  , can  be 
obtained  by  substituting  P^  and  P2f  into  Eqs.  (5.24)  and  (5.25).  Finally,  the  solution  of 

full-order  coupled  algebraic  Riccati  equation  and  full-order  state  weighting  matrices  can  be 
obtained  from  Eqs.  (5.26)  and  (5.22),  respectively. 

Recall  from  Theorem  5.1  in  Section  5.3.2  that  the  solution  of  the  LQR  controller  (Eq. 
(5.34))  belongs  to  the  solution  set  of  the  game  theoretic  controller  (Eq.  (5.31))  for  a single 
pole  shifting  case.  Therefore,  if  the  optimal  solution  to  the  optimization  criteria  on 
minimization  for  the  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  is 
equal  to  the  solution  to  the  LQR  controller,  then  the  optimal  solution  to  the  game  theoretic 
controller  is  equal  to  the  solution  to  the  LQR  controller;  if  the  optimal  solution  to  the  above 
optimization  criteria  is  the  not  equal  to  the  solution  to  the  LQR  controller,  then  the  optimal 
solution  to  the  game  theoretic  controller  is  better  than  the  solution  to  the  LQR  controller. 
Thus,  in  general,  the  solution  of  the  above  optimization  model  (KR1)  is  better  than  the 
solution  to  the  LQR  controller  in  the  sense  of  minimization  for  the  Frobenius  norm  of  the 
reduced-order  state  feedback  gain  matrix. 

Example  6. 1 : 


Consider  the  same  lateral  direction  model  of  the  F-4  aircraft  (Example  5.1)  in  Section 

The  controllable  system  is  given  as 

1 . 7680  0 . 4125  - 14 . 2500  0 . 0000' 

1 . 7440  8 . 9520' 

- 0 . 0070  - 0 . 3831  6 . 0380  0 . 0000 

x + 

- 2 . 9200  - 0 . 3075 

0.0016  - 0.9975  - 0.  1551  0.0586 

0 . 0243  - 0 . 0036 

u 

1 . 0000  0 . 0000  0 . 0000  0 . 0000 

0 . 0000  0 . 0000 

where  as  before 


x = A 


f roll  rate 
yaw  rate 

* sideslip 
roll  angle 


and  u 


(rudder 
aileron 
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The  quadratic  performance  indices  are 


00 


(xTQiX  + U^R^U^  + u2Ri2U2^  dt  ( i = 1,2) 


o 


The  input  weighting  matrices  are  selected  as  follow 

R j j =2,  /?12  = 0,  R2\  = 0,  and  R22  — 1 
Partitioning  the  control  task  results  in 


5,  = [1  . 7440  - 2 . 9200  0 . 0243  0 . 0000]7 
B2  = [8 . 9520  - 0 . 3075  - 0 . 0036  0 . 0000]7 


To  compare  the  game  theoretic  state  feedback  controllers  with  the  LQR  state  feedback 
controller,  the  LQR  performance  index  is  chosen  as 


The  open-loop  poles  for  the  above  system  matrix  of  the  F-4  aircraft  lateral  model  are 
[-  0 . 0150,  - 0 . 2149  ± 2 . 4858/,  - 1 . 8614],  These  poles  correspond  to  the  spiral 
mode,  the  Dutch  roll  mode,  and  the  roll  mode,  respectively.  Assume  that  the  control  task 
is  to  move  the  first  open-loop  pole  of  the  system  from  its  current  location  to  a new  location 
at  -0.5  (i.e.,  to  decrease  the  spiral  mode  time  constant)  and  to  minimize  the  Frobenius 
norm  of  the  reduced-order  state  feedback  gain  matrix.  Then,  the  solution  to  LQR  controller 
(Eq.  (5.34))  is 


The  corresponding  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  is 
0 . 1623.  Based  on  the  superimposed  optimization  criterion  that  minimizes  the  Frobenius 
norm  of  the  reduced-order  state  feedback  gain  matrix,  the  solutions  to  game  theoretic 
controller  (Eq.  (5.31))  is 


00 


0 

where  the  state  weighting  matrix,  Q,  and  the  control  weighting  matrix,  R,  are 

Q = Q\  + G2>  R = diag([Rn  R22\) 


Pr  = 0 . 0591 
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Px  = 0 . 1017,  P2  = 0 . 0508 

The  corresponding  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  is 
0 . 1570,  which  is  less  than  the  one  obtained  from  the  LQR  design. 

The  solution  set  to  game  theoretic  controller  and  the  solution  to  LQR  controller  are 
graphically  presented  in  Figure  6. 1 . Also  shown  in  Figure  6. 1 is  the  fact  the  LQR  controller 
belongs  to  the  solution  set  to  game  theoretic  controller. 

• : Solution  to  LQR  Controller 
O : Solution  to  Game  Theoretic  Controller 


Fig.  6. 1 The  solution  set  (solid  line)  to  game  theoretic  controller  and  the  solution  to 
LQR  controller. 

A time  domain  state  response  comparison  between  the  two  controllers  is  shown  in  Fig. 
6.2.  The  initial  conditions  for  the  states  are  x0  = [0  . 1,  0.1,  0.1,  0 . l]r.  Figures 
6.2(a),  6.2(b),  6.2(c),  and  6.2(d)  present  correspondingly  the  roll  rate,  yaw  rate,  sideslip  and 
roll  angle  state  responses  for  the  closed-loop  system  for  both  controller  controllers.  Figures 
6.2(e)  and  6.2(f)  show  the  rudder  and  aileron  responses.  Under  the  present  perturbed 
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(e)  (0 


Figure  6.2  LQR  controller  vs.  game  theoretic  controller  (Legend:  solid  line:  game 
theoretic  controller  (minimization  of  Frobenius  norm  of  Kr),  dash  line: 
LQR  controller  (minimization  of  Frobenius  norm  of  Kr). 
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configuration,  the  results  show  that  the  time  responses  of  the  roll  rate,  yaw  rate,  sideslip  and 
roll  angle  states  for  the  closed-loop  system  with  the  game  theoretic  controller  are  almost 
equal  to  their  corresponding  state  responses  for  the  closed-loop  system  with  the  LQR 
controller.  The  Frobenius  norm  of  u(t)  ( = [M](r)  u2(t)]T)  for  the  design  based  on  the  game 
theoretic  controller  are  less  than  the  one  for  LQR  controller.  For  example,  at  t = 0,  the 
Frobenius  norm  of  the  input  for  the  game  theoretic  controller  is  5 . 44  X 10  ~4 
(=  0 . 0122  + ( - 0 . 02)2);  the  Frobenius  norm  of  the  input  for  the  LQR  controller  is 

5 . 55  x 10  4 (=  0 . 0072  + ( - 0 . 0225)2).  The  numerical  result  agrees  with  that  the 
Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  using  a game  theoretic 
controller  design  is  less  than  the  one  using  a LQR  controller  design. 

6.2.2  Shifting  a Pair  of  Poles 

Given  an  n x n system  equation  (Eq.  (5.39)),  a pair  of  poles  shifting  problem  with 
minimum  square  of  the  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix  can 
be  constructed  using  the  following  optimization  problem  (KR2): 

Minimize  ||  Kr  ||jl 
subject  to 

Eqs.  (5.51)  and  (5.52) 

p\r  > °>  p2r  > 0.  (6.11) 

where 

_ \RnKpi'r 

Kr  [R22BT2rP2r 

□ 


First-order  and  Second-order  Necessary  Condition 
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The  Lagrangian  for  the  above  optimization  problem  (KR2)  is 


L =\ 


'w,; 

R22tB2I,2r 


ll/r  + Aj(mj  — «j)  + X2(m2  ~ ni) 


(6.12) 


where  mi  (i  =1,2)  are  defined  in  Eqs.  (5.46)  and  (5.47),  and  n;-  (i  =1,2)  are  defined  in 
Eqs.  (5.49)  and  (5.50).  According  to  Theorem  6.1,  the  first-order  necessary  condition  for 
the  optimization  problem  (KR2)  is 


^L(P\r  ,P2r  ) — 0 

* -j. 

where  (P\r  ,P2r)  is  a relative  minimum  point  of  the  optimization  problem  (KR2). 

Based  on  Theorem  6.2,  the  second-order  necessary  (sufficient)  condition  for  the 
optimization  problem  (KR2)  is  that  the  Hessian  matrix  of  the  Lagrangian  (Eq.  (6.12))  is 
positive  semi-definite  (definite). 

A MATLAB  code  in  the  Optimization  Toolbox,  “constr.m,”  is  used  to  solve  the 
non-linear  optimization  problem  (KR2).  Again,  once  the  solution  set,  P , and  P , , of  the 

i f Zf 

reduced-order  coupled  algebraic  Riccati  equation  is  obtained,  the  reduced-order  state 
weighting  matrices  glrand  Q2r  can  be  obtained  by  substituting  Plrand  P2  into  Eqs.  (5.24) 
and  (5.25),  then,  the  solution  of  full-order  coupled  algebraic  Riccati  equation  and  full-order 
state  weighting  matrices  can  be  obtained  from  Eqs.  (5.26)  and  (5.22)  respectively. 


Example  6.2: 

Consider  the  same  lateral  motion  model  of  the  F-4  aircraft  from  Example  6.1.  The 
open-loop  system  matrix  A and  the  control  influence  matrices  B,  Bx  and  B2  have  the  same 
value  as  they  did  in  Example  6.1.  The  input  weighting  matrices  are  given  as  Rn  = 1, 
Rn  = 0,  R2l  = 0,  R22  = 1,  and  R = diag{[  1 1]). 

The  open-loop  poles  for  the  system  matrix  of  the  F-4  aircraft  lateral  model  from 
Example  6.1  are  [-  0 . 0150,  - 0 . 2149  ± 2 . 4858;,  - 1 . 8614].  In  the  current 
example,  the  control  task  is  to  move  the  pair  of  complex  conjugate  poles  of  the  system  from 
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their  current  location  to  a new  location  at  — 1 ± 2 j.  This  is  equivalent  to  adding  more 
damping  to  the  Dutch  roll  mode. 

Numerical  results  for  the  full  state  feedback  gain  matrix  and  closed-loop  system  matrix 
are  provided  in  Table  6.1.  In  Tables  6. 1 , the  “game  theoretic  controller”  represents  the  state 
feedback  controller  based  a differential  game  design,  and  the  “LQR  controller”  represents 
the  state  feedback  controller  based  on  the  linear  quadratic  regulator  design. 

In  Table  6.1,  the  results  show  that  the  game  theoretic  controller  has  a smaller  ||  Kr  ||F 
than  the  LQR  controller.  Hence,  if  the  state  time  responses  for  both  closed-loop  systems  are 
very  close,  then  the  square  of  the  total  input  response  of  the  closed-loop  system  using  the 
game  theoretic  controller  should  be  smaller  than  those  for  the  closed-loop  system  using  the 
LQR  controller. 

Time  domain  state  response  comparisons  between  the  two  controllers  are  shown  in  Fig. 
6.3.  The  initial  conditions  for  the  states  are  x0  = [0  . 1,  0.1,  0.1,  0 . l]r.  Figures 
6.3(a),  6.3(b),  6.3(c),  and  6.3(d)  present  correspondingly  the  roll  rate,  yaw  rate,  sideslip  and 
roll  angle  state  responses  for  the  closed-loop  systems  using  the  game  theoretic  controller  and 
the  LQR  controller.  Figures  6.3(e)  and  6.3(f)  present  the  time  response  of  both  the  rudder 
and  aileron  inputs  for  both  closed-loop  systems.  The  graphical  results  show  that  the  time 
responses  for  the  roll  rate  state,  yaw  rate  state,  sideslip  state  and  rudder  input  are  about  the 
same  for  both  closed-loop  systems. 

However,  the  overshoot  percentages  of  time  responses  of  the  roll  angle  states  and 
aileron  input  for  the  closed-loop  system  utilizing  the  game  theoretic  controller  is  less  than 
those  for  the  closed-loop  system  using  the  LQR  controller.  These  results  are  again  in 
agreement  with  that  the  Frobenius  norm  of  the  reduced-order  state  feedback  gain  matrix 
obtained  by  a game  theoretic  controller  design  being  less  than  the  one  obtained  by  a LQR 
controller  design. 
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Table  6. 1 Resulting  Gains  and  Closed-loop  Matrices 


Game  Theoretic  Controller 


0 . 0095  - 0 . 5239  - 0 . 4232  0 . 0321 

0 . 0017  - 0 . 0609  - 0 . 1215  0 . 0040 


'-  1.7995  1.8718  - 12.4262  - 0.0919' 

0.0212  - 1 .9317  4.7650  0.0951 

ci  0 . 0014  - 0 . 9850  - 0 . 1453  0 . 0578 

1 . 0000  0 . 0000  0 . 0000  0 . 0000 

Kr  ||F  1 . 5272 


LOR  Controller 

0 . 0095  - 0 . 5208  - 0 . 4265  0 . 0320 

0 . 0019  - 0 . 0842  - 0 . 1 176  0 . 0053 


- 1 . 8017  2 . 0745  - 12 . 4539  - 0 . 1036' 

0 . 0212  - 1 . 9296  4 . 7566  0 . 0950 

0.0014  - 0.9851  - 0.1452  0.0578 

1 . 0000  0 . 0000  0 . 0000  0 . 0000 


1 . 5285 


6-3  Minimize  the  Envelop  Bound  of  the  Reduced-Order  State  Response 

In  this  section,  the  optimization  model  based  on  the  minimal  envelop  bound  of  the 
reduced-order  state  response  will  be  constructed  for  a pair  of  poles  shifting  case.  The 
following  derivation  will  not  be  applied  to  a single  pole  shifting  case,  since  the  time  response 
of  the  reduced-order,  1-by-l  system,  with  a prescribed  eigenvalue,  is  fixed. 

In  order  to  facilitate  future  derivations,  the  matrix  exponential  of  a 2 X 2 system  is  first 


reviewed. 
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Figure  6.3  LQR  controller  vs.  game  theoretic  controller  (Legend:  solid  line:  game 
theoretic  controller  (minimization  of  Frobenius  norm  of  Kr),  dash  line: 
LQR  controller  (minimization  of  Frobenius  norm  of  Kr). 
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Theorem  6.3  The  exponential  of  a general  2x2  matrix  (Bernstein  and  So  [70]) 
Let  X and  /u  denote  the  eigenvalues  of  A E C2  x 2,  then 


eA,  = ^r[(1  _ + 

II 

(6.13) 

a,  — Xe^1  pfo  _ Pnt 

eAt  = t—I  + ^ f -A, 

fi  — X /u  - X 

if n ^ X 

(6.14) 

Proof:  Refer  to  Reference  70. 


Theorem  6.3  and  Eq.  (5.39)  infer  that  Frobenius  norm  of  envelop  of  the  reduced  state 
response,  xr(t),  is  bounded  by  Frobenius  norm  of  Arc.  Hence  the  optimization  model  (AR2) 
is  defined  as  follow 


Minimize  ||  ATc  ||F 
subject  to 

Eqs.  (5.51)  and  (5.52) 

Plr  > 0,  p2r  > 0. 

where  Arc  is  defined  Eq.  (5.39). 


First-order  and  Second-order  Necessary  Condition 


(6.15) 


□ 


The  Lagrangian  for  the  above  optimization  problem  (AR2)  is 

1 = \\Arc  IIf  + - nx)  + X2(m2  - n2)  (6.16) 

where  Arc  is  defined  in  Eq.  (5.39),  mi  (z  =1,2)  are  defined  in  Eqs.  (5.46),  (5.47),  and  nt 

(i  =1,2)  are  defined  in  Eqs.  (5.49),  (5.50).  According  to  Theorem  6.1,  the  first-order 
necessary  condition  for  the  optimization  problem  (AR2)  is 

^L(PXr  ,P2r  ) = 0 

* £ 

where  (Plr  , P2r  ) is  a relative  minimum  point  of  the  optimization  problem  (AR2). 
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Based  on  Theorem  6.2,  the  second-order  necessary  (sufficient)  condition  for  the 
optimization  problem  (AR2)  is  that  the  Hessian  matrix  of  the  Lagrangian  (Eq.  (6.16))  is 
positive  semi-definite  (definite). 

The  solution  set  P,r  and  P2r  is  again  obtained  using  “constr.m”  in  MATLAB’s 
Optimization  Toolbox.  The  reduced-order  state  weighting  matrices,  g,r  and  Q2  , can  be 
obtained  by  substituting  Plf  and  P2f  into  Eqs.  (5.24),  (5.25).  The  solution  of  full-order 

coupled  algebraic  Riccati  equation  and  full-order  state  weighting  matrices  can  be  obtained 
from  Eqs.  (5.26)  and  (5.22),  respectively. 

Example  6.3: 

Consider  the  same  lateral  direction  model  of  the  F-4  aircraft  from  Example  6.2.  The 
open-loop  system  matrix  A,  the  control  influence  matrices  B,  B}  and  B2  and  the  input 
weighting  matrices  R,  R\\,  ^21  and  /?22  have  the  same  values  as  shown  in  Example 

6.2. 

The  open-loop  poles  for  the  system  matrix  of  the  lateral  motion  model  of  an  F-4  aircraft 
are  located  at  [ - 0 . 0150,  - 0 . 2149  ± 2 . 4858;,  - 1 . 8614],  The  controller  design 
task  is.  (1)  move  a pair  of  complex  conjugate  open-loop  poles  from  their  current  location 
to  the  new  location  at  [ — 1,  — 1 . 5],  (2)  move  two  real  open-loop  poles  from  their  current 
location  to  their  new  location  at  — 1 ± 2j,  (3)  minimize  the  envelop  bound  of  the 
reduced-order  state  response.  The  reason  for  moving  the  complex  poles  to  real  locations  and 
vice  versa  is  merely  to  demonstrate  the  methodology.  The  numerical  results  are  shown  in 
Tables  6.2  and  6.3.  In  Table  6.2,  a pair  of  complex  conjugate  poles  of  the  original  system 
matrix  are  shifted  to  the  location  at  [ — 1,  — 1 . 5].  The  full  state  feedback  gain  matrix  (A!)) 
and  the  intermediated  closed-looped  system  matrix,  AC[,  along  with  its  associated  Frobenius 
norm  of  the  reduced-order  system  matrix  A,c,  are  tabulated  in  Table  6.2.  Matrix  Ac  has  the 

two  new  real  poles  located  at  [ — 1 , —1.5]  and  the  two  original  real  poles  of  the  original 
open-loop  system  matrix  remains  unchanged. 
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Table  6.2  Gains  and  Closed-loop  Matrices  (Complex  Conjugate  Poles  to  Two  Real  Poles) 


Game  Theoretic  Controller 

0 . 0030 

- 0 . 7008  - 0 . 6545 

0 . 0385] 

0 . 1150 

3 . 5684  - 19 . 6990  - 

0 . 1365 

- 2 . 8023 

- 30.3097  160.9536 

1 . 1545] 

0 . 0370 

- 1 . 3322  1 . 8918 

0 . 0706 

0 . 0019 

- 0 . 9676  - 0 . 2419 

0 . 0572 

1 . 0000 

0 . 0000  0 . 0000 

0 . 0000 

K 


c,  IlF 


1 . 8577 


LOR  Controller 


Ky 


0 . 0054 

- 0 . 3383 

- 0 . 1801 

0 . 0204' 

0 . 1113 

- 0 . 1012 

- 13  . 8549 

0 . 0565 

- 2 . 7737 

1 . 9081 

110.0923  - 

0 . 5414" 

0 . 0429 

- 1 . 4021 

1 . 2516 

0 . 0770 

0 . 0019 

- 0 . 9896 

- 0 . 2006 

0 . 0583 

1 . 0000 

0 . 0000 

0 . 0000 

0 . 0000 

lrCl  IIF 


2 . 4707 


In  Table  6.3,  the  two  original  real  poles  of  the  intermediated  closed-loop  system  matrix 
are  shifted  from  their  current  locations  (i.e.,  - 0 . 0150  and  - 1 . 8614)  to  their  new 
locations  (i.e.,  — 1 ± 2j).  The  full  state  feedback  gain  matrix  ( K2 ),  and  the  final 
closed-loop  system  matrix,  AC2,  with  its  associated  Frobenius  norm  of  the  reduced-order 
system  matrix  Arc  are  also  listed  in  Table  6.3.  The  final  closed-loop  system  matrix,  ACl,  has 
a pair  of  complex  conjugate  poles  located  at  — 1 ± 2 j,  while  the  two  real  poles  (i.e., 
[ ~ — 1 . 5])  of  the  intermediated  system  matrix  remains  unchanged. 
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Table  6.3  Gains  and  Closed-loop  Matrices  (Two  Real  Poles  to  Complex  Conjugate  Poles) 


Game  Theoretic  Controller 


K2 


0.0051  0.0033 

0 . 0164  - 0 . 9369 


- 2 . 6470 
0 . 0172 
0 . 0020 
1 . 0000 


- 21  . 9280 

- 1 . 6107 

- 0 . 9711 
0 . 0000 


0 . 9476 
6 . 3016 


102 . 8892 
6 . 5965 
- 0 . 2423 
0 . 0000 


0 . 0032 
0 . 0869 


0 . 3710' 
0 . 1067 
0 . 0574 
0 . 0000 


3 . 2663 


LOR  Controller 


k2 


- 0.0050  0.0048  2.2237  0.0215' 

0 . 085  - 0 . 0081  - 3 . 801 1 - 0 . 0368 


- 2 . 8409  1 . 9726  140 . 2429 

0 . 0311  - 1 . 3907  6 . 5760 

0 . 0020  - 0 . 9898  - 0 . 2683 
1 . 0000  0 . 0000  0 . 0000 


- 0 . 2494' 

- 0 . 1285 
0 . 0577 
0 . 0000 


4 . 1769 


In  Tables  6.2  and  6.3,  the  “Game  Theoretic  Controller”  represents  the  state  feedback 
controller  based  on  a differential  game  design,  and  the  “LQR  Controller”  represents  the  state 
feedback  controller  based  on  a linear  quadratic  regulator  design.  Also  in  Table  6.2,  the 
subscript  “1”  represents  the  first  shifting  step  and  in  Table  6.3,  the  subscript  “2”  represents 
the  second  shifting  step. 

The  numerical  results  demonstrate  that  the  game  theoretic  controller  has  a smaller 
||f>  * = 1.2,  than  the  LQR  controller.  Therefore,  the  state  responses  for  the 
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closed-loop  system  using  the  game  theoretic  controller  should  have  tighter  response 
envelops  as  compared  with  the  LQR  design. 

The  initial  conditions  for  the  states  are  x0  = [0  . 1,  0.1,  0.1,  0 . I]7".  The  time 
responses  of  the  roll  rate,  yaw  rate,  sideslip  and  roll  angle  states  of  the  closed-loop  systems 
for  both  controllers  are  shown  in  Figures  6.4(a),  6.4(b),  6.4(c),  and  6.4(d),  respectively.  The 
rudder  and  aileron  state  time  responses  are  shown  in  Figures  6.4(e)  and  6.4(f), 
correspondingly. 

These  results  demonstrate  that  the  maximum  overshoots  of  time  responses  associated 
with  the  roll  rate,  yaw  rate,  sideslip  and  roll  angle  states  for  the  closed-loop  system  using  the 
game  theoretic  controller  are  less  than  those  maximum  overshoots  associated  with 
corresponding  state  responses  for  the  closed-loop  system  using  the  LQR  controller.  The 
results  satisfy  the  prediction  that  the  closed-loop  system  using  a game  theoretic  controller 
has  a narrower  envelop  of  the  state  time  response  envelop  than  the  one  using  a LQR 
controller. 

6.4  Minimize  the  Envelop  Bound  of  the  Full-Order  State  Response 

The  state  response  of  the  full-order  closed-loop  system  is 

x(t)  = eAc,x0  (6.17) 

where  Ac  is  the  full-order  closed-loop  system  matrix 

Ac  = A - | B^'BjP^T7  (6,18) 

(=1 

The  2-norm  of  the  state  response,  x(t),  can  be  expressed  as 

W0ll2  = ||^c^o|2 

^ Vc\i  Wl2  (6.19) 

The  bound  of  the  2-norm  of  the  state  response  depends  only  on  the  2-norm  of  matrix 

exponential  eAc  1 , since  the  2-norm  of  is  fixed  for  a give  perturbed  configuration  of  the 


rudder  [ini  sideslip  (rad)  roll  rate  [rad/sec! 
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(e)  (0 


Figure  6.4  LQR  controller  vs.  game  theoretic  controller  (Legend:  solid  line:  game 
theoretic  controller  (minimization  of  Frobenius  norm  of  Arc),  dash  line: 
LQR  controller  (minimization  of  Frobenius  norm  of  Arc). 
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linear  system.  The  following  theorem  is  used  to  determine  the  2-norm  of  a matrix 
exponential. 


Theorem  6.4  (Van  Loan  [63]) 


Given  a constant  matrix  A £ GnXn,  the  Schur  decomposition  states  that  there  exists  a 
unitary  matrix  Q E GnXn,  such  that 

QhAQ  = D + N (6.20) 

where  D is  a diagonal  matrix  and  Vis  a strictly  upper  triangular  matrix.  The  diagonal  entries 

of  D consist  of  system  eigenvalues  located  at  the  diagonal  entries.  The  2-norm  of  eAt  is 
bounded  by 


where 


< ea(A)  t ^ 
k= o 


|M|* 

k\ 


(6.21) 


a(A)  = max  { Re(A)  I X E A(A)}. 


(6.22) 


Proof:  The  poof  of  this  theorem  is  available  in  the  Reference  63. 

Based  on  theorem  6.4,  the  bound  of  the  2-norm  of  the  matrix  exponential  eAcl  is 
determined  by  the  2-norm  of  N,  since  a(A ) and  t are  fixed  for  a linear  system  with 
prescribed  eigenvalues  at  a specified  time  t. 

The  above  optimization  criterion  (i.e.,  the  2-norm  of  N)  can  be  superimposed  on  the 
constrain  equation  (Eq.  (5.31)  for  a single  pole  shifting  case,  or  Eqs.  (5.5 1 ),  (5.52)  for  a pair 
of  poles  shifting  case)  to  form  an  optimization  model  (AUC)  described  in  the  following 
section  for  “unconstrained”  full-order  state  feedback  gain  case.  Notice  that  the  term, 
“unconstrained,”  means  that  there  are  no  constraints  imposed  on  the  numerical  value  of  each 
individual  entry  in  the  state  feedback  gain  matrix. 


Unconstrained  Full-Order  State  Feedback  Gain  Case- 
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According  to  theorem  6.4  and  Eqs.  (6. 19),  (6.21),  the  2-norm  of  the  state  response,  x(t), 
is  bounded  by  2-norm  of  N.  Hence,  the  optimization  criterion  is  chosen  to  minimize  the 
2-norm  of  N to  obtain  a tighter  bound  of  full  state  response  envelop.  Based  on  that  criterion, 
the  optimization  problem  (AUC)  can  be  stated  as 

Minimize  ||//||2 
subject  to 

P - a ~ BlrRnlB^P]r  - B2R2^B2P 2r 
(or  Eqs.  (5.51)  and  (5.52)  for  shifting  a pair  of  poles) 

P\r  > 0.  P2r  > 0. 

□ 


First-order  and  Second-order  Necessary  Condition 


The  Lagrangian  for  the  above  optimization  problem  (AUC)  is 

_ 11^112  + ^i(«  ~ PBxRX][BTxPXr  - B2  R22lB2  P2r)  (single  pole) 

L — v.  (6  23) 

||  (V||2  + kx{mx  - nx)  + X2(m2  - n2 ) (a  pair  of  poles) 

According  to  Theorem  6. 1 , the  first-order  necessary  condition  for  the  optimization  problem 
(AUC)  is 

^L(P\r  >P2r  ) ==  0 

* * 

where  {P\r  ,P2r)  is  a relative  minimum  point  of  the  optimization  problem  (AUC). 

Based  on  Theorem  6.2,  the  second-order  necessary  (sufficient)  condition  for  the 
optimization  problem  (AUC)  is  that  the  Hessian  matrix  of  the  Lagrangian  (Eq.  (6.23))  is 
positive  semi-definite  (definite). 

The  solution  set,  P ^ and  P2  , of  the  non-linear  optimization  problem  (AUC)  is 
obtained  using  “constr.m”  in  MATLAB’s  Optimization  Toolbox.  The  reduced-order  state 
weighting  matrices,  Qlr  and  Q2 r,  can  be  obtained  by  substituting  Plr  and  P2  into  Eqs. 
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(5.24),  (5.25).  Finally,  the  solution  of  full-order  coupled  algebraic  Riccati  equations  and 
full-order  state  weighting  matrices  can  be  obtained  from  Eqs.  (5.26),  (5.22),  respectively. 

Recall  from  Theorem  5. 1 in  Section  5.3.2  that  the  solution  of  the  LQR  controller  (Eq. 
(5.34))  belongs  to  the  solution  set  of  the  game  theoretic  controller  (Eq.  (5.31))  for  a single 
pole  shifting  case.  Therefore,  if  the  optimal  solution  to  the  above  optimization  criterion  is 
not  equal  to  the  solution  to  the  LQR  controller,  then  the  optimal  solution  to  the  game  theoretic 
controller  must  be  better  than  the  solution  to  the  LQR  controller.  Thus,  in  general  the 
solution  of  the  above  optimization  model  (AUC)  (i.e.,  the  optimal  solution  to  the  game 
theoretic  controller)  is  better  than  the  solution  to  the  LQR  controller  in  the  sense  of 
minimization  of  the  2-norm  of  the  state  response  to  the  full-order  closed-loop  matrix. 

Example  6.4:  the  Unconstrained  Case 

Consider  the  same  lateral  direction  model  of  the  F-4  aircraft  from  Example  6.2.  The 
open-loop  system  matrix  (or  the  state  influence  matrix)  A,  the  control  influence  matrices 
B,  Bx,  B2  and  the  input  weighting  matrices  R,  Rn,  Rn,  R2l,  R22  all  have  the  same 
values  as  they  did  in  Example  6.2. 

The  open-loop  poles  for  the  system  matrix  of  the  lateral  motion  model  of  the  F-4  aircraft 
are  located  at  [ - 0 . 0150,  - 0 . 2149  ± 2 . 4858;,  - 1 . 8614],  The  control  task  is 

designed  to  move  the  open-loop  poles  from  current  location  to  [— 0 . 5,  — 1 ±2 j,  —2] 

(i.e.,  decrease  the  time  constants  of  all  the  modes  and  add  damping  to  the  Dutch  roll  mode), 
while  minimize  the  envelop  bound  of  the  full-order  state  response. 

The  numerical  results  are  shown  in  Tables  6.4  and  6.5.  In  Table  6.4,  a pair  of  complex 
conjugate  poles  of  the  original  system  matrix  are  shifted  from  their  current  location  to 
- 1 ± 2j.  The  full  state  feedback  gain  matrix  (Kx)  and  the  intermediated  closed-looped 
system  matrix  , AC{,  along  with  its  associated  2-norm  of  Ware  tabulated  in  Table  6.4.  Matrix 
AC]  has  a pair  of  complex  conjugate  poles  located  at  - 1 ± 2j,  while  the  two  real  poles  of 
the  original  open-loop  system  matrix  remain  unchanged. 
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Table  6.4  Resulting  Gains  and  Closed-loop  Matrices  (Complex  Conjugate  Poles) 


Game  Theoretic  Controller 


0 . 0080 
0 . 0122 


1 . 8913 

4 0 . 0202 
ci  0 . 0014 

1 . 0000 

|ATi»2 


0 . 0079 
0.0135 


1 . 9023 
0 . 0203 

ci  0 . 0015 

1 . 0000 

Nl2 


- 0 . 4979  - 0 . 2790  0 . 0301 

0 . 0183  - 1 . 5627  0 . 0046 


1 . 1 173  0 . 2262  - 0 . 0940’ 

- 1.8312  4.7428  0.0893 

- 0 . 9853  - 0 . 1539  0 . 0579 

0 . 0000  0 . 0000  0 . 0000 

4.1339 


LOR  Controller 


- 0 . 4837 

- 0 . 2899 

0 . 0293  ‘ 

- 0 . 0807 

- 1 . 5748 

0 . 0105 

1 . 9787 

0 . 3531 

- 0 . 1448' 

- 1 . 8204 

4 . 7073 

0 . 0888 

- 0 . 9860 

- 0 . 1537 

0 . 0579 

0 . 0000 

0 . 0000 

0 . 0000 

4.1615 


In  Table  6.5,  the  two  real  poles  of  the  intermediated  closed-loop  system  matrix  are 
shifted  from  their  current  location  (i.e.,  [ - 0 . 0150  - 1 . 8614])  to  the  new  location  (i.e., 
[-0.5  - 2]),  respectively.  The  full  state  feedback  gain  matrix  K2,  and  the  final 
closed-loop  system  matrix  ACl  (with  its  associated  2-norm  of  N)  are  also  listed  in  Table  6.5. 
The  final  closed-loop  system  matrix,  ACr  has  the  real  poles  located  at  - 0 . 5 and  - 2, 

while  the  pair  of  complex  conjugate  poles  of  the  intermediated  system  matrix  remain 
unchanged. 
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Table  6.5  Resulting  Gains  and  Closed-loop  Matrices  (Two  Real  Poles) 


Game  Theoretic  Controller 


k2 


0 . 0046  - 0 . 3923 
- 0.0082  0.1594 


- 0 . 3384 
0 . 1380 


- 2 . 5360 

- 0 . 9256 
0 . 0102 
1 . 0000 


1 . 1830 
- 1 . 8202 
- 0 . 9855 
0 . 0000 


- 0 . 5161 
3 . 6462 

- 0 . 1438 
0 . 0000 


- 0 . 6345 
0 . 2320 

- 1 . 0640' 

- 1 . 6922 
0 . 0741 
0 . 0000 


l*J2 


3.4656 


LOR  Controller 


k2 


- 0 . 1583  - 0 . 0526 
0 . 0587  0 . 0536 


- 0 . 0198 
0 . 0241 


- 2 . 0833 

- 0 . 0300 
0 . 0020 
1 . 0000 


1 . 7293 

- 2 . 2644 

- 0 . 9820 
0 . 0000 


- 0 . 0347 
4 . 5702 

- 0 . 1523 
0 . 0000 


- 0 . 4113 
0 . 1847 

- 1 . 0807' 

- 1 . 0553 
0 . 0686 
0 . 0000 


W2I2 


4.2328 


In  Table  6.4,  the  subscript  “1”  represents  the  first  shifting  step  and  in  Table  6.5,  the 
subscript  2 represents  the  second  shifting  step.  The  results  show  the  game  theoretic 
controller  has  a smaller  |AfJ2  (i  = 1,2)  than  the  LQR  controller.  Hence,  the  state  response 
for  the  closed-loop  system  using  the  game  theoretic  controller  should  result  in  a tighter 
response  envelop  compared  to  the  LQR  design. 

Figure  6.5  shows  the  state  responses  for  both  controllers.  The  initial  conditions  for  the 
states  are  [0,  0.1,  0.1,  0]r.  Figures  6. 5(a)-(d)  represent,  respectively,  the  roll  rate,  yaw 
rate,  sideslip  and  roll  angle  state  responses  for  the  closed-loop  system  (after  two  successive 
shifts)  for  both  controllers.  The  input  responses  are  shown  in  Figures  6.5(e)  and  6.5(f). 
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Figure  6.5  LQR  controller  vs.  game  theoretic  controller  (Legend:  solid  line:  game 
theoretic  controller  (minimization  of  full-order  state  response),  dash  line: 
LQR  controller  (minimization  of  full-order  state  response). 
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Based  on  the  current  perturbed  configuration,  the  results  show  that  the  time  responses 
for  all  the  sideslip  state,  yaw  rate  state,  rudder  input,  and  aileron  input  are  about  the  same 
for  both  closed-loop  systems.  However,  those  overshoots  associated  with  both  the  roll  rate 
and  roll  angle  state  responses  from  the  game  theoretic  controller  are  approximately  50%  of 
the  overshoots  of  their  associated  state  responses  generated  from  the  LQR  controller.  These 
results  are  in  agreement  with  the  ||  N2  ||2  results  in  Table  6.5. 

Up  to  this  point,  all  the  proposed  optimization  models  only  implement  pole  shifting 
tasks.  However,  as  mentioned  in  chapter  5,  it  is  possible  for  system  designers  to  add  other 
physical  or  “artificial”  design  specification  into  the  above  optimization  models.  These 
details  are  discuss  in  next  sub-section. 

Constrained  Full-Order  State  Feedback  Gain  Case.: 

Consider  a practical  design  situation  where  the  control  system  engineer  wants  to  modify 
an  existing  state  feedback  system  in  the  following  manner:  The  engineer  is  not  satisfied  with 
the  time  response  of  the  resulted  system.  He  wants  to  relocate  a single  pole  of  the  system 
without  changing  all  the  entries  of  the  full-order  state  feedback  gain  matrix  due  to  design 
considerations  (e.g.,  less  modification  effort,  or  physical  constraints  on  the  state  feedback 
gain  matrix).  Thus,  the  task  is  to  move  the  original  open-loop  pole  to  new  location,  while 
keeping  some  elements  of  state  feedback  gain  matrix  fixed. 

Based  on  this  assumption,  the  constrained  optimization  problem  (AC)  can  be  stated  as 


Minimize  ||iV||2 
subject  to 


M-a-  BirR-'BTplr  - B2R^BT2P2r 
(or  Eqs.  (5.5 1 ) and  (5.52)  for  shifting  of  a pair  of  poles) 


2rn  22  n2/2 


ij 
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The  quantity  ctj  in  the  above  model  is  a constant,  and  the  subscript  11  if  represents  the 
constrained  entry  located  at  Tth  row  and  y-th  column,  in  the  state  feedback  gain  matrix  ( K ). 

Notice  that  this  practical  design  problem  cannot  be  implemented  by  conventional  LQR 
pole  placement  technique,  since  the  LQR  pole  placement  methodology  yields  an  unique 
solution  as  shown  in  Section  5.3.2. 

Example  6.5:  the  Constrained  State  Feedback  Gain  Case 

In  this  example,  the  design  task  does  not  only  perform  the  single  pole  shifting,  but  also 
imposes  a state  feedback  gain  constraint.  The  open-loop  poles  for  the  above  system  matrix 
of  an  F-4  aircraft  lateral  model  are  [-  0 . 0150,  - 0 . 2149  ± 2 . 4858/',  - 1 . 8614], 
In  this  example,  it  is  assumed  that  there  exists  a designed  state  feedback  controller  which 
moved  the  first  open-loop  pole  of  the  system  from  its  open-loop  location  to  a location  at 
— 0.5  (Case  1 in  Table  6.6)  and  the  associated  state  feedback  gain  matrix  is  determined. 

However,  the  system  performance  is  not  satisfactory  and  it  is  desired  to  move  the  first 
pole  to  a new  location  at  - 1 , while  keeping  the  second  row  of  the  obtained  state  feedback 
gain  matrix  fixed  (Case  2 in  Table  6.6).  The  reason  for  keeping  the  second  row  of  state 
feedback  gain  matrix  unchanged  may  be  due  to  a geometric  constraint  on  the  aileron  angle. 

The  numerical  results  of  the  full  feedback  gain  matrix  ( K ),  the  closed-looped  system 
matrix,  Ac , along  with  its  associated  2-norm  of  N are  tabulated  in  Table  6.7.  In  Table  6.7, 
the  subscripts  “1”  and  “2”  represent  “Case  1”  and  “Case  2,”  respectively,  from  Table  6.6. 
The  constraint  of  the  optimization  model  for  the  game  theoretic  controller  of  Case  2 is 
fulfilled  as  shown  in  the  Table  6.7  (i.e.,  the  second  row  of  K2  is  the  same  as  the  one  of  A',). 

The  graphical  comparison  between  the  two  controllers  are  shown  in  Fig.  6.6.  The  initial 
conditions  for  the  states  are  x0  = [0,  0.1,  0.1,  0]r.  Figures  6.6(a),  6.6(b),  6.6(c),  and 
6.6(d)  present  the  time  responses  for  the  roll  rate,  the  yaw  rate,  the  sideslip  and  the  roll  angle 
states  of  the  closed-loop  system  for  both  controllers. 
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Table  6.6  Design  Specification  for  State  Feedback  Controllers  in  Example  6.5 


Case 

Method 

Closed-loop  Pole  Location 

Constraints  on  State 
Feedback  Gain 

1 

Nash 

[-  0 . 5,  - 0 . 2149  ± 2 . 4858;,  - 1 . 8614] 

No  Constraints 

2 

Nash 

[-  1 . 0,  - 0 . 2149  ± 2 . 4858;,  - 1 . 8614] 

State  Feedback  Gain 
(Aileron)  Same  as  Case  1 

The  graphical  results  show  that  the  overshoots  of  the  time  state  response  associated  with 
Case  1 (in  Table  6.6)  is  smaller  than  the  ones  with  Case  2 (in  Table  6.6).  It  agrees  with  the 
prediction  based  on  the  2-norm  of  N.  The  rudder  input  for  Case  2 is  greater  than  the  one  for 
Case  1,  and  the  aileron  input  for  the  Case  2 is  smaller  than  the  one  for  Case  1.  This 
phenomena  is  due  to  the  constraint  on  the  state  feedback  gain  matrix  for  Case  2. 

Extension  to  the  /V-Player  Differential  Games 

Previous  numerical  examples  (i.e..  Examples  6. 1 to  6.5)  has  demonstrated  the 
capability  of  shifting  either  a single  pole  or  a pair  of  poles  from  their  current  location(s)  to 
any  location  in  the  s-plane  for  the  2-player  cases.  As  mentioned  in  Chapter  5,  the  proposed 
methodology  can  be  easily  extend  to  solve  an  rc-player  differential  game.  The  following 
example  will  be  used  to  the  demonstrate  this  extensity. 


Example  6.6: 


Consider  a longitudinal  motion  model  of  an  AIRC  aircraft  [71].  The  controllable 
system  is  given  as 


x = Ax  + Bu 

'0 . 0000  0 . 0000  1 . 1320 
0 . 0000  - 0 . 0538  - 0 . 1712 
0 . 0000  0 . 0000  0 . 0000 
0 . 0000  0 . 0485  0 . 0000 

0 . 0000  - 0 . 2909  0 . 0000 


(6.24) 

0 . 0000  - 1 . 0000' 

0 . 0000  0 . 0705 

1 . 0000  0 . 0000 
- 0 . 8556  - 1 . 0130 
1 . 0532  - 0 . 6859 


A = 
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Table  6.7  Resulting  Gains  and  Closed-loop  Matrices 


Game  Theoretic  Controller  (1st  pole  from  -0.015  to  -0.51 


0 . 0000 

0 . 0000 

0 . 0000 

0 . 0000' 

0 . 0589 

0 . 1384 

- 0 . 0267 

0 . 1043 

- 2 . 2955 

- 0 . 8268 

- 14 . 0108 

- 0 . 9337' 

0 . 0111 

- 0 . 3405 

6 . 0298 

0 . 0321 

0 . 0018 

- 0 . 9970 

- 0 . 1552 

0 . 0590 

1 . 0000 

0 . 0000 

0 . 0000 

0 . 0000 

IWJ2  14.8640 


Game  Theoretic  Controller  (1st  pole  from  -0.015  to  -1  and 
aileron  feedback  gain  fixed) 


- 0 . 0975  - 0 . 2292  0 . 0442  - 0 . 1727 

0.0589  0.1384  - 0.0267  0.1043 


- 2 . 1252 

- 0 . 2737 
0 . 0042 
1 . 0000 


- 0 . 4267  - 14 . 0880  - 0 . 6323' 

- 1 . 0097  6 . 1589  - 0 . 4721 

- 0 . 9914  - 0 . 1563  0 . 0632 

0 . 0000  0 . 0000  0 . 0000 


IN2 


14.7826 


0 . 0000 

0 . 0000 

0 . 0000" 

- 0 . 1200 

1 . 0000 

0 . 0000 

0 . 0000 

0 . 0000 

0 . 0000 

4 . 4190 

0 . 0000 

- 1 . 6650 

1 . 5750 

0 . 0000 

- 0 . 0732 

The  state  and  input  variables  are  defined  as 
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X 10  1 X |Q-1 


Figure  6.6  Legend:  Solid  line:  game  theoretic  controller  (1st  pole  from  -0.015  to 
-0.5),  dashed  line:  game  theoretic  controller  (1st  pole  from  -0.015  to  -1 
with  2nd  row  of  K2  given) 
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X = 1 

Z 

u 

d 

> - , 

altitude  [m] 
forward  speed  [m/sec] 
pitch  angle  [degrees] 

q 

w 

pictch  rate  [deg/sec] 

vertical  speed  [m/sec] 

- > 

u 


spoiler  angle  [degrees/ 10] 

< forward  acceleration  [m/sec2]  > 
elevator  angle  [degrees] 


The  quadratic  performance  indices  are 


h = J (xrQpc  + u\Raux  + uT2Ri2u2  + w[/?;3w3)  dt  (i  =1,3) 
o 

The  input  weighting  matrices  are  selected  as  follow 

^11  = ^12  = 0,  /?  1 3 = 0, 

/?2i  = o,  r22  = 1,  r22  = o, 

/?3i  = 0,  R22  = 0,  /?33  = 1 
Partition  of  the  control  task  results  in 

5,  = [0 . 0000  - 0 . 1200  0 . 0000  4 . 4190  1 . 5750]7 

B2  = [0 . 0000  1 . 0000  0 . 0000  0 . 0000  0 . 0000] T 

fl3  = [o  . 0000  0 . 0000  0 . 0000  - 1 . 6650  - 0 . 0732]7 

In  order  to  compare  a game  theoretic  state  feedback  controllers  with  a LQR  state  feedback 
controller,  the  compatible  performance  index  is  chosen  as 

OO 

J = JX+J2  + J2  = ^ ( xtQx  + utRu ) dt. 

0 

where  the  state  weighting  matrix,  Q,  and  the  control  weighting  matrix,  R,  are 
Q = Q\  + Q2  + C?3>  R = diag{[Rn  R22  /?33]) 

The  open-loop  poles  for  the  above  system  matrix  of  the  aircraft  model  are  located  at 
[-  0 . 7801  ± 1 . 0296y,  - 0 . 0176  ± 0 . 1826y,  0 . 0000].  These  poles  are  associated 
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with  the  short  period,  phugoid  period,  and  altitude,  respectively.  Assume  that  the  control 
task  is  to  move  the  last  open-loop  pole  (located  at  0 . 0000)  of  the  system  from  its  current 
location  to  a new  location  at  -0.5  and  to  minimize  the  envelop  bound  of  full-order  state 
time  response.  The  solution  to  LQR  controller  (Eq.  (5.34))  is 


In  3-player  game,  let  a be  a single  pole  of  the  linear  system  which  is  to  be  moved  from 
its  current  location  to  its  new  location,  say  a.  The  associated  equality  constraint  equation 
for  a single  pole  shifting  case  is 


Based  on  the  superimposed  optimization  criterion  that  minimizes  the  envelop  bound 
of  the  full-order  state  response,  the  solutions  to  game  theoretic  controller  (Eq.  (6.25))  is 


The  2-norm  of  N associated  with  both  controller  design  are  listed  in  Table  6.8.  The 
results  show  the  game  theoretic  controller  has  a smaller  \\N\\2  than  the  LQR  controller.  Hence, 
the  state  response  for  the  closed-loop  system  using  the  game  theoretic  controller  should  have 
a tighter  response  envelop  as  compared  to  the  LQR  design. 

The  time  response  comparisons  between  the  two  controllers  are  shown  in  Fig.  6.7.  The 
initial  state  conditions  are  *0  = [0,  1,  0,  0,  2]r  Figures  6.7(a),  6.7(b),  6.7(c),  6.7(d), 
and  6.7(e)  represent  correspondingly  the  altitude,  forward  speed,  pitch  angle,  pitch  rate  and 
vertical  speed  state  responses  for  the  closed-loop  system  for  both  controllers.  Figures  6.7(f), 
6.7(g)  and  6.5(h)  show  the  spoiler,  forward  acceleration,  and  elevator  responses. 

Under  the  current  perturbed  configuration,  the  results  show  that  the  time  response  of 
all  the  state  variables  for  the  game  theoretic  controller  have  less  overshoots  than  the  one  for 
the  LQR  design.  Particularly,  the  overshoots  associated  with  altitude,  forward  speed  and 
pitch  angle  state  responses  for  game  theoretic  controller  are  only  50%  of  the  associated 

responses  for  the  LQR  controller.  This  results  agree  with  the  II  N h results  in  Table  6.8. 


Pr  = 2.  1037 


3 


(6.25) 


P\r  = 3.8651,  P2r=  1.8779,  P3(.  = 48 . 4925 
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Table  6.8  Design  Specification  for  State  Feedback  Controllers  in  Example  6.6 


Controller 

N 2 

Superimposed  Optimization  Criteria 

NASH 

1.6633 

Minimization  of  envelop  bound  of  full-order  state  time  response 

LQR 

2.1037 

- 

6.5  Summary 

The  inverse  procedure  proposed  to  solve  differential  game  problems  has  a numerical 
advantage  over  the  regular  approach,  since  it  does  not  require  the  solution  of  the  complex 
coupled  algebraic  Riccati  equations.  Three  optimization  models  are  proposed  as  additional 
optimization  criteria:  (1)  Minimum  square  of  the  Frobenius  norm  of  the  reduced-order  state 
feedback  gain  matrix,  (2)  Minimize  the  envelop  bound  of  reduced-order  state  time  response, 
and  (3)  Minimize  the  envelop  bound  of  full-order  state  time  response. 

The  proposed  optimization  methodology  not  only  performs  both  single  pole  and  a pair 
of  poles  shifting,  but  also  has  the  capacity  to  incorporate  other  (different  from  the  chosen 
superimposed  optimization  criterion)  controller  design  specifications. 

A lateral  motion  model  of  an  F-4  aircraft  is  used  to  demonstrate  the  procedure.  Both 
unconstrained  and  constrained  state  feedback  controller  are  considered  for  the  minimal 
envelop  bound  of  the  full-order  state  time  response.  For  the  unconstrained  case  (i.e.,  no 
constraints  on  the  individual  entries  of  the  state  feedback  gain  matrix),  the  numerical  results 
show  that  the  closed-loop  system  using  a game  theoretic  controller  has  less  overshoot  than 
the  closed-loop  system  using  a LQR  controller.  This  is  in  agreement  with  the  proposed 
optimization  criterion.  For  the  constrained  case,  the  design  task  can  only  be  implemented 
with  a game  theoretic  controller. 

A longitudinal  motion  model  of  an  AIRC  aircraft  with  three  active  controllers  is  used 
to  demonstrate  the  extensile  of  the  proposed  procedure.  The  results  show  that  this  procedure 
can  be  extended  to  solve  the  pole  placement  problems  for  the  general  n-player  games. 


107 


Figure  6.7  LQR  controller  vs.  game  theoretic  controller  (notation  : solid  line:  game 
theoretic  controller  (minimization  of  full-order  state  response),  dash  line: 
LQR  controller  (minimization  of  full-order  state  response). 


CHAPTER  7 
SUMMARY 

7.1  Summary  of  Results 

In  this  research,  two  methodologies  for  designing  a linear  time  invariant  game  theoretic 
state  feedback  controller  in  the  regulator  problem  are  proposed.  First,  given  a set  of  state 
and  input  weighting  matrices  and  their  associated  integral  quadratic  performance  indices, 
integrative  schemes  based  on  Homotopy  theory  are  utilized  to  solve  the  resulting  coupled 
algebraic  Riccati  equations.  To  improve  the  efficiency  of  the  algorithm,  the  symmetric 
property  of  Riccati  matrices  is  explored  in  order  to  reduced  the  number  of  the  unknowns  to 
be  solved.  Second,  given  a set  of  desired  closed-loop  pole  locations  and  the  input  weighting 
matrices  in  the  integral  quadratic  performance  indices,  an  inverse  methodology  for 
designing  a linear  time  invarient  full  state  feedback  game  theoretic  controller  is  developed. 

In  the  forward  design  process  (i.e.,  integral  quadratic  performance  indices  and  their 
associated  input  and  state  weighting  matrices  are  given),  the  developed  Homotopy  algorithm 
for  an  algebraic  Riccati  equation  has  two  advantage  over  existing  Homotopy  code: 

(1)  The  developed  algorithm  determines  the  positive  definite  solution  of  an  algebraic 
solution  without  the  need  of  finding  all  the  possible  solutions. 

(2)  The  developed  algorithm  is  considered  computationally  efficient,  since  it  utilized 
the  symmetric  property  of  the  solution  of  an  algebraic  Riccati  equation  in 
conjunction  with  the  Kronecker  sum. 

For  coupled  algebraic  Riccati  equations,  the  proposed  algorithm  is  also  very  efficient 
computationally  (in  comparison  with  other  homotopy  algorithms).  Numerical  examples 
demonstrate  that  the  proposed  algorithm  can  solve  problems  that  a Newton-iterative 
algorithm  cannot  solve. 
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In  the  inverse  procedure,  a new  optimal  pole  placement  methodology  based  on 
differential  game  theory  for  a general  multiobjective  linear  regulator  problem  is  proposed 
and  developed.  By  using  multiple  integral  quadratic  performance  indices,  the  control 
domain  can  be  expanded  and  thus  allow  the  control  system  designer  the  freedom  of 
implementing  additional  criteria  to  specify  the  system’s  performance.  The  developed 
method  is  capable  of  shifting  either  a single  pole  or  a pair  of  poles  (two  real  poles  or  a pair 
of  complex  conjugate  poles)  to  the  desired  closed-loop  location(s)  while  satisfying 
additional  design  specifications  for  the  linear  system.  Multiple  poles  shifting  tasks  can  be 
implemented  by  utilizing  the  shifting  algorithm  from  a single  pole  or  a pair  of  poles 
recursively.  It  is  found  there  are  several  advantages  to  the  proposed  methodology. 

( 1 ) .  The  proposed  methodology  can  determine  state  feedback  controllers  in  an  n-player 

differential  game  without  actually  solving  n-coupled  algebraic  Riccati  equation 
numerically, 

(2) .  The  proposed  method  can  incorporate  a design  specification  (e.g.,  minimize  the 

envelop  bound  of  full-order  state  time  response)  as  an  optimization  criterion. 

(3) .  The  proposed  methodology  is  better  than  conventional  inverse  LQR  scheme. 

(4) .  The  proposed  methodology  can  incorporate  other  controller  design  considerations 

(e.g.,  constraints  on  the  elements  of  the  state  feedback  gain  matrices)  as  additional 
equality  or  inequality  constraints  for  this  optimization  problem. 

7.2  Suggestions  for  Future  Work 

Some  of  the  results  of  this  research  are  open-ended.  For  example,  the  Homotopy 
algorithm  for  coupled  algebraic  Riccati  equations  may  not  lead  to  positive  definite  solutions, 
although  the  author  has  not  experienced  any.  Some  areas  where  research  can  be  invested 
with  immediate  benefit  are  outlined  below. 

(1).  Modify  the  current  Homotopy  algorithm  for  the  coupled  algebraic  Riccati 
equations  to  ensure  the  obtained  solution  is  positive  definite,  which  is  the  necessary 
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condition  for  the  state  feedback  game  theoretic  controller  to  stabilize  the 
closed-loop  system. 

(2) .  For  the  inverse  procedure,  other  design  specifications  can  be  considered  as 

superimposed  optimization  criteria  for  different  dynamical  systems  (i.e.,  the  set  of 
criteria  presented  in  Chapter  6 is  not  exhaustive). 

(3) .  The  proposed  design  of  game  theoretic  controller  is  based  on  the  Nash  strategy, 

there  are  few  researches  on  the  other  game  theoretic  strategies  (i.e.,  Mini-Max, 
Stackelberg,  Pareto  strategies)  has  rarely  found  in  literature,  especially  through  the 
inverse  procedure.  Thus,  it  is  of  interest  to  conduct  research  on  these  strategies. 

(4) .  The  current  research  is  for  the  deterministic  models  based  on  the  time  domain 

analysis,  the  stochastic  process  and/or  frequency  domain  analysis  could  also  be  a 
rich  area  of  research. 

(5) .  In  this  research,  the  analytical  model  for  a system  is  assumed  exact.  However,  in 

a real  control  problem,  there  always  exists  the  difference  between  an  analytic 
model  and  a “true”  model.  Thus,  the  study  on  the  issue  of  the  “robust”  pole 
placement  technique  for  the  multiobjective  optimization  problems  should  be 
carried  on  to  make  the  current  research  results  applicable  to  the  real  control 
problems. 


APPENDIX:  POLE  PLACEMENT  BY  LINEAR  QUADRATIC  REGULATOR 


The  content  of  this  appendix  is  based  mainly  on  Arar  and  Sawan’s  work  [59],  However, 
the  notation  is  modified  for  the  purpose  of  clarity.  A correction  of  Arar  and  Sawan’s  paper 
on  the  optimization  criterion  for  a pair  of  pole  shifting  is  also  made. 

A1  Mathematical  Model  for  Linear  Quadratic  Regulator 

A linear  controllable  and  observable  dynamical  system  which  is  mathematically 
modelled  in  a state  equation  form  is, 

x(t)  = Ax(t ) + Bu(t ) (Al) 

where  u(t)  C represents  control  input  to  the  plant,  and  x(t)  €E  <R>n  represents  the  prefect 
measured  plant  state.  The  control  law,  u,  are  restricted  to  state  feedback  (i.e.,  u(x,  t )). 

Matrices  A and  B are  appropriately  dimensioned.  The  performance  index  is  defined  as 

00 

J = j (xtQx  + uT Ru)  dt  (A2) 

o 

where  Q are  positive  semi-definite  matrices,  and  R are  positive  definite  matrices.  The  state 
feedback  controllers,  u,  are 

u(t)  = - R~lBTPx(t ) (A3) 

where  P is  the  positive  definite  solution  of  the  following  algebraic  Riccati  equation 

Q + PA  + AtP  - PBR~xBtP  = 0 (A4) 

Here,  we  assume  that  a set  of  closed-loop  eigenvalues  is  given  and  a control  weighting 
matrix,  R,  is  pre-chosen.  Arar  and  Swan’s  scheme  [59]  is  used  to  find  state  weight  matrices, 
Q,  such  that  the  system  poles  are  shifted  to  the  desired  locations. 


Ill 
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A2  Pole  Placement  hv  LOR  Controller 


A2.1  Preliminary  Transformation 


Consider  the  following  transformation 

xr  = TTx.  (A5) 

where  x E 9k”  represents  the  plant  states,  e 9kr  represents  the  “reduced”  plant  states, 
and  rG  3V!  x r are  the  reduced  left  eigenvectors  of  the  open-loop  matrix,  A , associated  with 
the  open-loop  poles  which  will  be  shifted.  Apply  this  transformation  to  Eq.  (Al)  and  yield 
the  reduced  system  equation 

MO  = Arxr(t)  + Brur(t).  (A6) 

where  Ar  G <$>rXr  and  Br  G %rXm  are 

TT A = ArTr.  (A7) 

Br  = TtB  (A8) 

The  reduced  performance  index  associated  with  Eq.  (A2)  is 

00 

Jr  = I (xjQrxr+ujRur)  dt.  (A9) 

o 

where  Qr  G 9^>rXr  is 

Q = TQrTT  (A  10) 

The  reduced  state  feedback  control  law,  ur,  of  the  reduced  system  is 

MO  = ~ R~]BjPrxr(t)  (All) 

Pr  is  the  positive  definite  solution  of  the  following  reduced  algebraic  Riccati  equation 

Qr  + PAr  + Ajpr  - PrBrR~xBTrPr  = 0 (A  12) 

where 


P = TPrTr. 

The  reduced  state  feedback  gain  matrix  is 


(A13) 
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Kr  = R lBrPr.  (A  14) 

The  reduced  closed-loop  system  resulting  from  the  reduced  control  law  becomes 

Mt)  = (A  r-  BrKr)Xr(t).  (A  15) 

Substitution  of  Eqs.  (A5),  (A7),  (A8),  (A13)  and  (A14)  intoEq.  (A15),  the  full  order  model 
of  the  close-loop  system  becomes 

x(t)  = (A  - BK)x(t).  (A  16) 

where 

K = KrTT  = R~'BjPrTT  (A  17) 

A2.2  Shifting  a Single  Pole 


Let  a single  pole,  a,  of  the  linear  system  be  moved  from  a current  location  to  a new 
location,  say  p.  Choose  T to  be  the  left  eigenvector  associated  with  the  left  eigenvalue,  a, 
of  the  system,  then  the  reduced  system  matrix  in  Eq.  (A  15)  is 

p = a - BrR~]BjPr.  (A18) 

The  scalar  Pr  can  be  solved  as 


_ (a  - p) 
r BrR~' Br- 

and the  state  weighting  matrix  that  will  shift  a system  pole  form  X to  p is 

Qr=  ~ 2 XPr  + PrBrR~xBTrPr. 


(A  19) 


(A20) 


A2.3  Shifting  a Pair  of  Poles 


Let  a pair  of  the  left  eigenvalues,  a + /3 'j , £ — ftj,  of  a n x n system  matrix  A,  with 
associated  non-zero  left  eigenvectors,  u ± jv  (or  u and  v for  the  two  real  poles  shifting 
case),  to  be  shifted  from  current  location  to  a new  location  at  y + Sj,  (p  — & j,  where 

£ = a,  if/3  * 0 

£ * a,  if/?  = 0 
and 


(p  = y,  if  (3  ^ 0 
(p  ^ y,  if  (5  = 0 
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The  transformation  matrix,  T,  is  chosen  as 


T = 


Notice  that  the  above  transformation  matrix  is  only  for  a pair  of  complex  conjugate  poles 
shifting  case.  For  the  case  of  two  real  poles  shifting,  the  transformation  matrix  still  has  the 
same  form  as  shown  in  the  above  equation,  u and  v,  however,  are  the  left  eigenvectors  of 
the  system  matrix  corresponding  to  the  two  real  eigenvalues. 

From  Eqs.  (A5),  (A  14),  (A  15),  apply  the  above  transformation  matrix  to  Eq.  (Al)  and 
obtain  the  reduced  system  equation 


Arc  Xj{t) 

where  Arc  is  the  reduced  closed-loop  system  matrix 

Arc  = Ar  - BrR~XBTrPr. 
The  quantities  Ar  and  Pr  are  defined  as 


(A21) 


a 1 3 

-p  z 


(A22) 


Pr 


X Z 

z y . 


(A23) 


and  the  reduced  closed-loop  system  matrix,  Arc.  In  Arar  and  Sawan’s  paper  [59],  the 
following  assumptions  are  made:  (1)  u G 9*>2  and  (2)  the  control  weighting  matrix  R is 
equal  to  diag([\  1]).  Based  on  these  assumptions. 


(A24) 


BrR~lBj  = [ a.  b 
b c 

The  reduced  closed-loop  system  matrix,  A rc,  can  be  further  derived  and  it  has  a characteristic 
polynomial  equation  as  follow 


s + mxs  + m2  = 0 


where 


(A25) 
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mx  = hxx  + h^y  + h2z  + h4 

(A26) 

m2  = f\Z2  ~ f\xy  + fa  + f3y  + f4z  + f5 

(A27) 

with 

l j,  h2  l2,  h3  = 2 l3,  h4  — q — £ 

(A28) 

l}  = a2  + b2,  l2  = c2  + d2,  l3  = ac  + bd 

(A29) 

f\  h f2  ~ Ph  f?>  ~ ~ Ph  ~ ah  (A30) 

Pa  = Ph  ~ ah  ~ %h  ~ Ph*  fs  = P2  + 

(A31) 

Since  Arc  has  a complex  conjugate  (left)  eigenvalue  pairs,  y + dj,<p  - 
equation  of  the  reduced  system  matrix,  Arc,  can  be  also  written  as 

dj,  the  characteristic 

s 2 + n^s  + n2  = 0 

(A32) 

where 

n\  = - 2 y ( - y - (p,  if  (5  = 0) 

n2  = y2  + d2  (y(p,  if  d = 0) 

By  equating  Eq.  (A25)  with  Eq.  (A32),  we  get 

mx  = «! 

(A33) 

m2  — n2 

(A34) 

Equations  (A33)  and  (A34)  are  the  only  equality  constraint  equations  for  this  problem. 
However,  we  have  three  unknowns  (i.e.  x,  y,  and  z)  to  be  solved.  Since  the  solutions  of 
Eqs.  (A33)  and  (A34)  are  not  unique,  we  have  to  superimpose  an  optimization  criteria  on 

this  problem. 


A3  Correction  for  the  Ill-defined  Optimization  Model 

In  Reference  59,  Arar  and  Sawan  proposed  an  constraint  optimization  model  (LI)  for 
the  case  of  shifting  of  a pair  of  pole  as  follow 

Min  ( trace  Pr ) 
subject  to 
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Eqs.  (A33)  and  (A34) 

Pr  > 0,  (i.e.,  x > 0 and  xy  — z2  > 0) 

The  Largrangian  for  the  optimization  model  (LI)  is 

L = x + y + Xxx{hxx  + htf  + h3z  + hA  — nx)  + 

h<f\z2  ~ f\xy  + fa  + fa  + fa  +f5-  n2) 
The  gradient  of  L will  give  the  first  order  necessary  condition  as 

1 + Xxhx  + X2(f2  - fxy)  = 0 
1 + Xxh2  + X2(f3  — fxx)  = 0 
Xxh3  + X2(2 fxz  +/4)  = 0 

The  Hessian  matrix  of  L is 


□ 


(A35) 


(A36) 


H(L)  = 


0 


A/, 

0 


A/ 1 0 

0 0 
0 2A^ i 


(A37) 


which  has  eigenvalues  of  [ A^j,  A^,  2A^ j ].  Recall  that  the  second  order  necessary 


condition  for  any  constraint  optimization  problem  is  that  the  Hessian  matrix  of  the 
Largrangian  is  positive  semi-definite.  Clearly,  the  Hessian  matrix  in  Eq.  (A37)  is  positive 
semi-definite  if  and  only  if  /,  = 0,  since  X2  will  not  be  zero.  From  Eqs.  (A29)  and  (A30), 
we  get 


f\  = ~ ( ad  - be)2  (A38) 

This  implies  that  /,  can  be  zero  if  only  the  determinant  of  Br  is  zero.  Hence,  if  the 
determinant  of  Br  is  not  zero,  which  is  true  for  the  most  general  case,  the  numerical  solution 
for  the  optimization  model  (LI)  is  not  a relative  minimum. 


Correction 


There  are  many  design  criteria  for  a linear  control  system,  e.g.,  the  criterion  stated  in 


Sec.  6.1  can  be  used  as  a superimposed  optimization  criterion.  Here,  the  author  use  a 
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different  criterion,  which  is  “minimum  of  the  square  of  Frobenius  norm  of  solution  to  the 
reduced  algebraic  Riccati  equation.”  The  reason  for  choosing  this  as  an  additional 
optimization  criterion,  is  to  make  the  symbolic  derivation  feasible.  Base  on  the  new 
proposed  criterion,  a pair  of  poles  shifting  problem  can  be  re-constructed  as  the  following 
optimization  model  (L2) 


Min  ||  Pr  ||2 
subject  to 

Eqs.  (A33)  and  (A34) 

Pr  > 0,  (i.e.,  x > 0 and  xy  - z2  > 0) 


The  Largrangian,  L2,  for  the  optimization  model  (L2)  is 

^2  = x1  + y2  + 2z2  + Xll(hlx  -I-  h^y  + h3z  + /z4  - nx)  + 

*n(f\Z2  - f\*y  + /2*  + h y + U + f5  - n2 ) 

The  gradient  of  L2  will  give  the  first  order  necessary  condition  as 

2x  + A j j/z  j + Aj2(/2  — f\y)  = 0 
2y  + An h2  + X]2(f3  - fxx)  = 0 
4z  + A nh3  + A12(2 fxz  +/4)  = 0 

The  Hessian  matrix  of  L2  is 


□ 


(A39) 


(A40) 


H(L2)  = 


2 - A12/j  0 

^•l/l  2 0 

0 0 4 + 2A,/, 


(A41) 


which  has  the  eigenvalues  of  [2  - Xx2fx,  2 + A12/j,  4 + 2Aiafi].  Hence,  the  second  order 
necessary  condition  is 


-2  <A,/,  < 2 
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